Recovering Files From an NTFS Drive

Important lessons for computers owners:

  1. Set up a backup system
  2. Use that backup system regularly

It turns out that while I set up a backup system for my wife’s computer, the last time it was ran was sometime late 2009… So when the primary Windows partition on that computer no longer booted; was not recognised as a Windows install by the Windows install CD rescue option; and would not mount using the ntfs-3g driver from Linux, I knew I was in trouble! Luckily, I had split the Windows install to have a separate partition where most of her data (photos, music, documents) were sitting and that partition was still mountable. So, in the worst case scenario, the total damage would be limited to Firefox and Thunderbird profiles and whatever was saved on the desktop.

Recovering more data from the hard-drive was not particularly difficult as the hard-drive still basically worked. In that case, the first step in any data recovery attempt is to create a disk image of the broken partition. This prevents any further damage to the hard-drive making you life worse. A good tool for the job is GNU ddrescure (which may be named ddrescue or gddrescue depending on your Linux distribution). Make sure you have the GNU version as the other software with the same name is less user friendly. It can take a long time to take an image of a hard-drive so having some nice output and being able to stop and restart at any stage are quite essential. Creating an image is as simple as running (as root):

ddrescue -r3 /dev/sda2 imagefile logfile
The log file shows that the hard-drive had quite a few corrupt parts, all relatively small and close together so my guess is some sort of physical damage preventing those being read.

# Rescue Logfile. Created by GNU ddrescue version 1.13
# Command line: ddrescue -r3 /dev/sda2 imagefile logfile
# current_pos current_status
0x30D31E00 +
# pos size status
0x00000000 0x309EB000 +
0x309EB000 0x00001000 -
0x309EC000 0x00073000 +
0x30A5F000 0x00002000 -
0x30A61000 0x00073000 +
0x30AD4000 0x00002000 -
0x30AD6000 0x00073000 +
0x30B49000 0x00002000 -
0x30B4B000 0x000FC000 +
0x30C47000 0x00001000 -
0x30C48000 0x00074000 +
0x30CBC000 0x00001000 -
0x30CBD000 0x00074000 +
0x30D31000 0x00001000 -
0x30D32000 0x7F5409E00 +

So now you have an image of your hard-drive, it is time to get those files out. Given the disk image was relatively complete, I went for software called Sleuth Kit. It has a frontend called Autopsy, which I found fairly useless apart from browsing the data. To use it, start Autopsy by pointing it at directory where all its files are to be stored and then point your web browser at the relevant place:

mkdir autopsydir
autopsy -d autopsydir
firefox http://localhost:9999/autopsy

While browsing the data was good to confirm that most of the files I wanted were still there, I just wanted to extract every single file possible from the image and I would then copy all the relevant stuff over to the new computer as necessary. Slueth Kit has some command-line tools for doing that; fls for listing files and icat for getting them. Using these, you can extract the files one at a time… but a simple script will automate extracting everything.

#!/bin/sh
IMAGE=../imagefile
fls -urp $IMAGE |
while read type inode name; do
   echo $name
   case $type in
      d/d) mkdir -p "$name" ;;
      r/r) [ ! -f "$name" ] && icat $IMAGE $(echo $inode | sed 's/://g') > "$name" ;;
   esac
done

That will take a long time to run, especially if it is your primary Windows install drive as it will have lots of small files to extract. So I made that script so that it can continue where it left off. Just do not stop one run in the middle of an important file or it will not finish extracting it. You might want to also touch hiberfil.sys and pagefile.sys in the extraction directory first as they are relatively useless and will take up a few gigabytes each.

So everything appears to have been recovered and I survived the lack of backup! Windows 7 has a good feature that reminds you to set up a backup and run it regularly, so hopefully I will not need to do this again.

Posted in Software on by Allan Comments Off

GCC in C++

As is becoming widely covered, the GCC Steering Committee and the FSF have approved the use of C++ in the GCC codebase. This is not a particularly sudden decision… I originally saw this proposed by Ian Lance Taylor on his blog a couple of years ago. He also has some good slides about how using C++ would be benificial. There was a gcc-in-c++ branch that corrected incompatibilities flagged by -Wc++-compat, but I think this is mostly merged and there is now an experimental --enable-build-with-cxx configure flag. So I think that this decision comes at no real surprise to anyone involved.

I think this is a great idea! Why? Because if the compiler is written in C++, then the compiler developers have more motivation to make C++ compilation faster. This is good for me as C++ is my primary choice for a writing in a compiled programming language. So this is a win for me.

Is it a win for GCC? I know some people (especially Linus Torvalds) think using C++ for anything is a major disaster. In fact, despite being a C++ proponent, I tend to agree… 99% of people who propose the usage of C++ for something are wrong. Many of the complexities in C++ have no place in most projects and too many C++ programmers feel the need to use the entire C++ toolset. Let be honest, the curiously recurring template pattern and template metaprogramming have no real place anywhere but in academia[1]. But (single) inheritance and the STL do provide what I have seen people try to replicate in C many times. Using C++ as a C with classes is not really that different from C but it can be much simpler to write.

There are some obvious cases where changing to C++ in the GCC codebase would be of great benefit. Take a look in gcc/vec.h in the gcc source.

/* The macros here implement a set of templated vector types and
associated interfaces. These templates are implemented with
macros, as we're not in C++ land. The interface functions are
typesafe and use static inline functions, sometimes backed by
out-of-line generic functions. ...

That is screaming out to be replaced by a std::vector. There are other examples where simple inheritance is mimicked using a slew of (un)defines and switch statements. Some of these are so complex, I wonder whether there will be any performance loss due to the introduction of virtual function calls. Certainly, it will be a win in terms of maintainability.

[1] Although in combination you get the expression template paradigm, which allows you to build a really nice numeric vector class that unrolls all loops at compile time and does not suffer from virtual function overhead, making it as fast as manually programming the vector arithmetic in C but much more convenient to use. Then you go back to using std::valarray which is close enough…

Thunderbird vs Exchange IMAP

I was having issues with attachments only being partially downloaded in Thunderbird from an IMAP account on a MS Exchange server (2003 I think). Turns out I ran into a very old Thunderbird bug (filed 2001-07-24). MS Exchange by default does not return the actual size to the IMAP4 command “FETCH RFC822.SIZE”. This is deliberate as it provides a nice perfomance advantage.

So… like a good Linux user I should once again blame Microsoft software for being broken? Not this time. It turns out section 3.4.5 of RFC 2683 (IMAP4 Implementation Recommendations) says that the RCF822.SIZE value should only used for things like providing estimates to the user and not for allocating buffers and the like. Bad Thunderbird!

The work around is to go to Preferences > Advanced > Config Editor… and set mail.server.default.fetch_by_chunks to FALSE. Old emails will need to be redownloaded to fix the attachments.

Simple MacBook Pro Fan Daemon

The fan control on the MacBook Pro under Linux is not the best… I would say it does not work at all but I once saw the fan speed increase slightly on its own so it appears to do something, sometimes, according to some logic I can not figure out.

It turns out that taking “manual” control of the fan is quite easy. A simple

echo 1 > /sys/devices/platform/applesmc.768/fan1_manual
sets the fan to manual mode. Then you can adjust fan1_output in the same directory to set the current fan speed. Do not get confused with fan1_input, as that is strangely the actual fan speed! The minimum and maximum speeds are given by fan1_min and fan1_max. If the minimum speed is reported as 0 or 1, ignore it. For MacBook Pros with Core 2 Duo processors the minimum fan speed should be 2000. The maximum fan speed is 6200.

So now that we know how to control the fan, we just need some sort of algorithm to choose what the fan speed should be based on the temperature. The MacBook Pro has a whole bunch of temperature sensors, but the ones that matter are for the processors as they are always the highest. These are found in /sys/devices/platform/coretemp.{0,1}/temp1_input (you may need to load the coretemp module). Montoring these during basic usage shows the average temperatures of the two processors is around 40-45C during idle, 50-55C with basic web browsing and 60-65C when watching a HD movie (at least in warm Australian ambient temperatures).

To save battery on a laptop, I think that the fan should not come on when the computer is doing anything less intensive than watching a movie, so I set that fan to kick in at 65C. This coincides with what Mac OSX does. From OSX, it appears that the fans should hit full speed at 80C and the speed builds up exponentially to that point. The formula I use for changing the fan speed when the temperature is increasing is:

temp <= 65:
   speed = max(current_speed, 2000)
65 < temp < 80:
   step = (6200 - 2000) / ( (80 - 65) * (80 - 64) / 2 )
   speed = max(current_speed, ceil(2000 + (temp - 65) * (temp - 64) / 2 * step))
temp >= 80:
   speed = 6200

When the temperature is decreasing, I prefer to keep the fan going slightly longer to force the temperature down to low levels as quickly as possible. I push it back down to 55C using this formula:

temp >= 80:
   speed = 6200
55 < temp < 80:
   step = (6200 - 2000) / ( (80 - 55) * (80 - 54) / 2 )
   speed = min(current_speed, floor(6200 - (80 - temp) * (81 - temp) / 2 * step)
temp <= 55:
   speed = 2000

Here is a graphic of what that looks like (red = increasing, blue = decreasing):

fan speed

Grap the source code here. It assumes two processors and a single fan (not true for all MacBook Pros…). For Arch Linux users, there is also a PKGBUILD and daemon (mbpfan) for ease of use. I am lazy, so there is very little error checking in the code. It works for me but use at your own risk…

Update (2011-08-11): MBP Fan Daemon Update

Stupid SDL_mixer

SDL_mixer is my least favourite piece of software today and is being very closely followed by SDL_image. A few updates that should have taken me about half an hour today ended up taking over three hours due to bugs in these packages.

I had a bunch of SDL related packages (sdl_gfx, sdl_image, sdl_mixer, sdl_perl) to update. These are usually fun, because they require extensive game playing for testing! My first point of testing is always Chromium B.S.U. Everything looked good there so I upload the updated sdl_gfx and sdl_image packages. Now for the second line of testing: Frozen Bubble. Interesting… SDL_mixer has an soname bump despite being an update from 1.2.9 to 1.2.10. Stranger things have happened (heimdal likes slipping soname bumps into minor version releases). So a rebuild of many other packages is in order. Rebuild frozen-bubble and try again…

FATAL: Couldn't load '/usr/share/frozen-bubble/gfx/loading.png' into a SDL::Surface.

Hmm… looks like an SDL_image issue. I can confirm this using some other games too. It turns out the SDL_image 1.2.9 release is very broken; just not broken enough to break Chromium B.S.U. So much for my extensive testing… It appears fixed in upstream SVN, but I am feeling lazy and just downgrade it while waiting for the new release that appears to be due in the next day.

Now, try Frozen Bubble again. Still no good…

[Graphics..........] [Sound Init] at /usr/games/bin/frozen-bubble line 312

Looks like the SDL_mixer update is causing issues. That may not be surprising given the soname bump. This is confirmed by starting the game with the --no-sound option. While searching for a fix, I discovered that the soname bump was an accident. Apparently, this has been fixed, but either the source was missed or something else was going on. The good news is, I do not have to do a big rebuild now (but the rebuilds I had already done were a waste). The bad new is, SDL_mixer is still broken… The change-log in SVN lists this:

Fixed bug loading multiple music files

Pull that patch and we can all play games again! Luckily, not all updates are that painful.

Edit: updates are now available for both SDL_mixer and SDL_image that fix these issues.

Posted in Software on by Allan Comments Off

Taking Moblin For A Spin

I thought I should take the Moblin v2 Beta release for a spin to see what all the excitement is about. While most distros would run fine on a netbook with little modification (e.g. The EEE repo for Arch essentially just contains a kernel), Moblin started from scratch and provides an interface specifically designed to get the most out of netbooks. Also, it is optimized for Intel’s Atom processor, so should provide a boost to multimedia applications on those less powerful processors that netbooks contain (in particular optimizations for the SSE2 and SSE3 instruction sets).

I was intending to test the Moblin Beta (20090529) in VirtualBox as I do not actually have a netbook, but that is not supported yet. While the live CD will boot, the screen resolution is too small for the upper menu to fit and the functionality is severely impaired. It did boot and run well on my laptop (any computer with recent Intel parts should be fine), but I was limited to testing from the live CD due to lack of spare partition to install it on.

My initial impressions are all positive. The interface makes it easy to do what I would want to do on a netbook (browse the internet and watch videos). Connecting to wireless internet was simple and sound worked out of the box. The web browser and media players did their job. The concept of “zones”, rather than multiple desktops, also works well. I would not be expecting to run too many applications at the same time on a reasonably underpowered machine anyway. I would need more time to assess how useful “the myzone” actually is, but my initial impress is that it could use improvement.

A couple of things did frustrate me:

  • The inability to configure the top menu. I do not use Twitter or Last.fm so it would be good to get rid of the “status panel”.
  • The “myzone” page can only have its background customized. This seems an area where more configuration options would be useful.
  • I spent a lot of time looking for a shut-down button. It appears hitting the power button on the laptop does a shutdown (without a confirmation dialog…).
  • The lack of a system tray. An instant messager is a bit useless without one.

I am sure some/all of these will be addressed as the distro heads towards an actual release. Only not being able to configure the top menu and the lack of system tray are critical as far as I am concerned.

Overall, I will definitely consider Moblin quite favorably if/when I get a netbook.

Posted in Software on by Allan Comments Off

Posting (with) Drivel

It does not take long to realize that posting from within WordPress is not the most efficient process. For me, two posts was enough… So, the hunt was on to find a desktop application to post with.

Gnome Blog is in the Arch repos, but is a bit basic. The main feature missing for my purposes was the ability to add a category to posts.

A search in the AUR found Drivel. It has several useful features, including the ability to add a category to posts (however, only one per post and not tags), being able to edit recent posts and integrated spell-checking. Even better, its development is now being continued after a several year break.