Software

Converting Video On The Command Line

I recently acquired an iPad for work purposes… so the most important thing to know is how to convert video to play on it. Use handbrake, done – short blog post.

But, that would be all too simple. Often I want to watch an entire season of a show that I have collected from various sources over the years and these often have widely varying sound levels. That is quite annoying if you set the season to play and then have to adjust the volume for every episode.

Here is a simple guide to convert your videos into a format suitable for the iPad with equalized volumes. I somewhat deliberately used a variety of tools for illustration purposes, but I think the one I selected tended to be the quickest for each step. The following code snippets assume that your videos have extension “.avi”. They also destroy source files, so make a backup.

Step 1. Extract the audio track using mplayer:
for i in *.avi; do
  mplayer -dumpaudio "$i" -dumpfile "${i//avi}mp3"
done

Step 2. Make sure all audio is mp3 and convert using ffmpeg if not:
for i in *.mp3; do
  if ! file "$i" | grep -q "layer III"; then
    mv "$i" "$i.orig"
    ffmpeg -i "$i.orig" "$i"
    rm "$i.orig"
  fi
done

Step 3. Normalize the audio levels using mp3gain:
mp3gain -r *.mp3

Step 4. Stick the adjusted audio back into the video file using mencoder (part of mplayer):
for i in *.avi; do
   mv "$i" "$i.orig"
  mencoder -audiofile "${i//avi}mp3" "$i.orig" -o "$i" -ovc copy -oac copy
  rm "$i.orig" "${i//avi}mp3"
done

Step 5. Convert to iPad format using the command line version of handbrake:
for i in *.avi; do
  HandBrakeCLI -i "$i" -o "${i//avi/m4v}" --preset="iPad"
  rm "$i"
done

This is probably not the most efficient way of doing this and will become less so once handbrake can normalize volume levels by itself (which appears to be being somewhat worked on by its developers…). But when you have several seasons of a show, each with more than 50 episodes (only a few minutes each), you quickly become glad to be able to make a simple script to do the conversion automatically.

Posted in Software on April 19th, 2012 by Allan – 2 Comments

Getting My Touchpad Back To A Usable State

I was happy to note the following in the release announcement of xf86-input-synaptics-1.5.99.901:

“… the most notable features are the addition of multitouch support and support for ClickPads.”

As a MacBook Pro user, that sounded just what I needed. No more patching the bcm5974 module to have some basic drag-and-drop support. I upgraded and everything seemed to work… for a while. I began noticing weird things occurring when I was trying to right and middle click (being a two and three finger click respectively). Specifically, my fingers would sometimes be registered in a click and sometimes not.

The two finger “right” click was easy enough to figure out. If my fingers were too far apart, the two finger click was not registered. It turns out I am what I have decided to call a “finger-spreader”, as my fingers can be quite relaxed across the touchpad when I click. Fair enough, I thought… I just have to train myself to click with my fingers closer together. Then came the three finger click. All three fingers close together did not seem to register as anything. A bit of spreading and a three finger click got registered, but too much spreading and it was down to two fingers. Also, it was not actual distance between fingers that mattered as rotating my hand on the touchpad with my fingers the same distance apart could result in different types of clicks registered.

A bit of looking in the xf86-input-synaptics git shortlog lead me to this commit as a likely candidate for my issues. The commit summary starting with “Guess” and it having a comment labelled “FIXME” were the give-aways… The first thing I noticed was that the calculation of the maximum distance between fingers to register as a multitouch click was done in terms of percentage of touchpad size. That means that the acceptable region where your two fingers need to be for a two finger click is an ellipse, which at least explains why physical distance appeared not to matter.

Attempted fix #1 was to increase the maximum allowed separation between fingers from 30% to 50%. That worked brilliant for two finger clicking, but made three finger clicking even worse, which lead me to another interesting discovery… The number of fingers being used is calculated as the number of pairs of fingers within the maximum separation, plus one. For two fingers, fingers 1 and 2 form one pair, plus one is “two fingers”. However, for three fingers, there are three possible pairs: (1,2), (2,3) and (1,3). This explains the weirdness in three finger clicking; finger pairs (1,2) and (2,3) must be withing the maximum allowed separation while finger pair (1,3) must be outside that. That explains why having fingers too close together did not register as a three finger click (as it was being reported as four fingers – three pairs plus one) and why things became worse when I increased the maximum allowed separation. I filed a bug report upstream and a patch to fix that quickly appeared.

After applying that patch, multifinger clicks all work fine provided your fingers are close enough together. I do not find the required finger closeness natural so I got rid of the finger distance restrictions altogether using this patch. I am not entirely sure what I break removing that, but it appears to be something I do not use as I have not noticed an issues so far. As always, use at your own risk…

Posted in Software on April 11th, 2012 by Allan – 6 Comments

How Secure Is The Source Code?

With the addition of source code PGP signature checking to makepkg, I have began noticing just how many projects release their source code without any form of verification available. Or even if some form of verification is provided, it is done in a way that absolutely fails (e.g. llvm which was signed by a key that was not even on any keyservers meaning it could not be verified). If code security fails at this point, actually signing packages and databases at a distribution end-point instills a bit of a false sense of security.

To assess how readily validated upstream source code is, I did a survey of what I would consider the “core” part of any Linux distribution. For me, that basically means the packages required to build a fairly minimal booting system. This is essentially the package list from Linux From Scratch with a few additions that I see as needed…

For each source tarball I asked the following questions: 1) Is a PGP signature available and is the key used for signing readily verified? 2) Are checksum(s) for the source available and if they are only found on the same server as the source tarball, are they PGP signed? The packages are rated as: green – good source verification; yellow – verification available but with concerns; red – no verification. Apologies to any colour-blind readers, but the text should make it clear which category each package is in…

Package Verification
autoconf-2.68 PGP signature, key ID in release announcement, key readily verifiable.
automake-1.11.3 PGP signature, key used to sign release announcement.
bash-4.2.020 PGP signature for release and all patches, link to externally hosted key from software website.
binutils-2.22 PGP signature, key used to sign release announcement (containing md5sums).
bison-5.2 PGP signature, key ID in release announcement, externally hosted keyring provided to verify key.
bzip2-1.0.6 MD5 checksum provided on same site as download.
cloog-0.17.0 MD5 and SHA1 checksums in release announcement posted on off-site list.
coreutils-8.15 PGP signature, key used to sign release announcement.
diffutils-3.2 PGP signature, key used to sign release announcement.
e2fsprogs-1.42.1 PGP signature, key readily verifiable.
fakeroot-1.18.2 MD5, SHA1 and SHA256 checksums provided in PGP signed file, key readily verifiable.
file-5.11 No verification available.
findutils-4.4.2 PGP signature, link to externally hosted key in release announcement.
flex-2.5.35 No verification available.
gawk-4.0.0 PGP signature, key difficult to verify.
gcc-4.6.3 MD5 and SHA1 checksums provided in release email. MD5 checksum provided on same site as download.
gdbm-1.10 PGP signature, key ID in release announcement (with MD5 and SHA1 checksums), key readily verifiable.
gettext-0.18.1.1 PGP signature, key readily verifiable.
glibc-2.15 No release tarball, download from git (PGP signature available when release tarball is made).
gmp-5.0.4 PGP signature, key ID and SHA1 and SHA256 checksums on same site as source, key difficult to verify otherwise.
grep-2.11 PGP signature, key used to sign release announcement.
groff-1.21 PGP signature, key difficult to verify.
grub-1.99 PGP signature, key used to sign release announcement.
gzip-1.4 PGP signature, key used to sign release announcement.
iana-etc-2.30 No verification available.
inetutils-1.9.1 PGP signature, key readily verifiable.
iproute-3.2.0 PGP signature, key readily verifiable.
isl-0.09 No verification available.
kbd-1.15.3 File size available in file in same folder as source.
kmod-0.05 PGP signature, key readily verifiable.
less-444 PGP signature, key posted on same site as download, key difficult to verify otherwise.
libarchive-3.0.3 No verification available.
libtool-2.4.2 PGP signature, key readily verifiable, MD5 and SHA1 checksums in release email.
linux-3.2.8 PGP signature, key readily verifiable.
m4-1.4.16 PGP signature, key used to sign release announcement.
make-3.82 PGP signature, key used to sign release announcement.
man-db-2.6.1 PGP signature, key used to sign release announcement.
man-pages-3.35 PGP signature, key readily verifiable.
mpc-0.9 (libmpc) PGP signature, key readily verifiable.
mpfr-3.1.0 PGP signature, key readily verifiable.
ncurses-5.9 PGP signature, key used to sign release announcement.
openssl-1.0.0g PGP signature, key readily verifiable.
pacman-4.0.2 PGP signature, key readily verifiable.
patch-2.6.1 PGP signature, key difficult to verify.
pcre-8.30 PGP signature, key readily verifiable.
perl-5.14.2 MD5, SHA1, SHA256 checksums provided on same site as download.
pkg-config-0.26 No verification available.
ppl-0.12 PGP signature, key readily verifiable.
procps-3.2.8 No verification available.
psmisc-22.16 No verification available.
readline-6.2.002 PGP signature for release and all patches, link to externally hosted key from software website.
sed-4.2.1 PGP signature, key difficult to verify.
shadow-4.1.5 PGP signature, key readily verifiable.
sudo-1.8.4p4 PGP signature, key difficult to verify.
sysvinit-2.88 PGP signature, key difficult to verify.
tar-1.26 PGP signature, key used to sign release announcement.
texinfo-4.13a PGP signature, key difficult to verify.
tzdata-2012b Many checksums provided in release announcement.
udev-181 PGP signature, key readily verifiable.
util-linux-2.21 PGP signature, key readily verifiable.
which-2.20 No verification available.
xz-5.0.3 PGP signature, key difficult to verify.
zlib-1.2.6 MD5 checksum provided on same site as download (although download mirrors available).

Note that some of these packages have additional methods of verification available (e.g. those that are PGP signed may also provide checksums and file sizes), but I stopped looking once I found suitable verification. When I label a key as “readily verifiable”, that means it is either signed by keys I trust, that it is used to sign emails that I can find or it is posted on the developers personal website (which must be different from where the source code is hosted). I personally found my preferred method of verification was packages whose release announcements were signed by the same key as the source.

While you might look at that table and think there is a lot of green (and yellow) there so everything is in reasonable shape, it is important to note that the majority of these are GNU software and all GNU software is signed. Also, 15% of the packages in that list have no source verification at all. From some limited checking, it appears the situation quickly becomes worse as you move further away from this core subset of packages needed for a fairly standard Linux system, but I have not collected actual numbers to back that up yet.

Posted in Software on April 7th, 2012 by Allan – 3 Comments

MBP Fan Daemon Update

For those using my simple MacBook Pro fan daemon, you probably want to check that it still works… At least on my system, the location of the core temperature measurements have changed from /sys/devices/platform/coretemp.{0,1}/temp1_input to /sys/devices/platform/coretemp.0/temp{2,3}_input. I think this occured with the update to Linux 3.0 (but I am too lazy to confirm that is the actual update to blame…).

If you also have this change, you can grab an updated version of the daemon here. As always, it is only tested on my machine (MBP 5.5 13″), so it may not work anywhere else without adjustment…

Posted in Software on August 11th, 2011 by Allan – 1 Comment

Syncing Files Across SFTP With LFTP

My webhost only provides SFTP access (which is not surprising given what I pay…). But this can become annoying for maintaining things like a package repository where I would like to keep the remote files in sync with my local copy. My first thought was to go with a FUSE based solution in combination with rsync. Looking into the current best options to mount the remote directory (probably sshfs), I was eventually lead to LftpFS and on to its underlying software LFTP.

LFTP is a sophisticate command-line file transfer program with its own shell-like command syntax. This allows syncing from my local repo copy to the remote server in a single command:

lftp -c "open -u <user>,<password> <host url>; mirror -c -e -R -L <path from> <path to>"

The -c flag tell LFTP to run the following commands (separated by a semicolon). I use two commands; the open command (should be obvious what it does…) and a mirror command. The only real “trick” there is to add -L to the mirror command, which makes symlinks be uploaded as the files they point to. This is required as the FTP protocol does not support symlinks and repo-add generates some.

That was exactly what I needed and it makes a nice bash alias being a single command.

Posted in Software on July 29th, 2011 by Allan – 5 Comments

Local WordPress Install On Arch Linux

After the WordPress update from 3.1.3 to 3.1.4 unexpectedly broke one of the plugins I use (My Link Order – why this was removed as a native feature in WordPress is beyond me…), I decided it was time to actually test updates locally before I pushed them to my site. That would also allow me to locally test theme changes and new plugins rather than just doing it live and attempting to quickly revert all breakages I made. It is still not the worlds best testing set-up as it does not use the same web server, PHP or MySQL version as my host, but I am fairly happy assuming the basics of WordPress will be compatibly with what my host provides and so only really need to test functionality that should not be affected by such differences.

Note I decided to go with Nginx as the web server as it seemed an easy way to go. I also did not use the WordPress package provided in the Arch Linux repos as it kind of defeats the whole purpose of testing the upgrade, requires slightly more set-up in nginx.conf and I think files in /srv/http should not be managed by the package manager (but that is another rant…).

So here is a super-quick ten-step guide to getting a local WordPress install up and running.

  • pacman -S nginx php-fpm mysql
  • Adjust /etc/nginx/conf/nginx.conf to enable PHP as described here
  • Enable the mysql.so and mysqli.so extensions in /etc/php/php.ini
  • sudo rc.d start mysqld php-fpm nginx
  • If this is your first MySQL install, run sudo mysql_secure_installation
  • Give yourself permission to write to /srv/http/nginx
  • Download and extract the WordPress tarball into /srv/http/nginx
  • Create the MySQL database and user as described here
  • Adjust the wp-config.php file as needed (see here)
  • Point your browser at http://127.0.0.1/wp-admin/install.php

And it is done! I have not attempted to set-up the auto-update features in WordPress as that involves either setting up and FTP or SSH server and I have no need to do either on my laptop.

As a bonus, I can now draft blog posts while offline and preview them with all their formatting. So you can all look forward to more rambling posts here from me…

Posted in Software on July 25th, 2011 by Allan – 3 Comments

The “python2″ PEP

When Arch Linux switched its /usr/bin/python from python-2.x to python-3.x, it caused a little controversy… There were rumours that it had been decided upstream that /usr/bin/python would always point at a python-2.x install (although what version that should be was unclear). Although these rumours were abundant and so more than likely such a discussion did occur (probably offline at PyCon 2009), this decision was never documented. Also, whether such a decision can formally be made off the main development list is debatable.

Enter PEP 394. Depending on how I am feeling, I call this the “justify Arch’s treatment of python” PEP or the “make Debian include a python2 symlink” PEP. Either way, the basic outcome is:

  • python2 will refer to some version of Python 2.x
  • python3 will refer to some version of Python 3.x
  • python should refer to the same target as python2 but may refer to python3 on some bleeding edge distributions

The PEP is still labeled as a draft, but all discussion is over as far as I can tell and I think it will probably be accepted without much of any further modification. The upshot is, using “#!/usr/bin/env python2” and “#!/usr/bin/env python3” in your code will become the gold standard (unless of course you code can run on both python-2.x and python-3.x). There is still no guarantee what versions of python-2.x or python-3.x you will get, but it is better than nothing…

One recommendation made by the PEP is that all distribution packages use the python2/python3 convention. That means the packages containing python-3.x code in Arch should have their shebangs changed to point at python3 rather than python. Given our experience doing the same thing with python2, this should not be too hard to achieve and is something that we should do once the PEP is out of draft stage. This has a couple of advantages. Firstly, we will likely get more success with upstream developers preparing their software to have a versioned python in their shebangs (or at least change all of them when installing with PYTHON=python2 ...). That would remove many sed lines from our PKGBUILDs. Secondly, if all packages only use python2 or python3, then the only use of the /usr/bin/python symlink would be interactively. That would mean that a system administrator could potentially change that symlink to point at any version of python that they wished.

Posted in Software on March 24th, 2011 by Allan – 7 Comments

Python 3 Module Support

I was reading a thread on the python-dev mailing list where someone asked about ideas for porting modules to python-3.x. That lead me to wonder what the current status of python-3 support was in various well used modules. So here is a summary of my research. As a disclaimer, I have not directly asked any of the involved module developers what their opinion is, but have rather summarised what I can find posted on various mailing lists. Also, the modules I list are my own subjective selection of what I think are popular modules, so do not take inclusion/exclusion in this list to mean anything.

GUI Toolkits:

PyQt – Has supported python-3 since v4.5 (release 2009-06-05).

PyGTK – Is effectively going to be a dead project with the upcoming release of GTK+3 so will not be ported to python-3. Instead, projects should move to PyGObject, which has some limited python-3 support with v2.26.0 (base and Introspection modules only – released 2010-09-27)

wxPython – python-3 support has not been started and according to their roadmap will likely be on hold until a major rewrite is finished.

Science:

numpy – Supports python-2 and python-3 from a single code base since v1.5.0 (released 2010-08-31), although the module needed for its testing framework (python-nose) is not yet officially ported.

scipy – Will support python-3 with the release of v0.9.0. As far as I can tell, python-3 support is finished in their SVN.

matplotlib – I could not find any details on the current status of python-3 support for matplotlib, so I assume this has yet to start. The porting of numpy to python-3 should give this a huge boost.

Database:

pysqlite – the sqlite module has been part of the Python stdlib for a while now, so has supported python-3 from the beginning. I guess the external development version has similar support.

psycopg2 – This is actually quite a weird situation… It appears to have been ported at the end of 2008 and looking at the patch should have been able to simulataneously support python-2 and python-3. However, when it got applied to the master git branch in mid 2009, a python2 branch was immediately created and all development for the last 18 months has occurred on that branch.

mysql-python – An old blog post indicates there is a rewrite happening and python-3.x support will be added once this starts shaping up to beta level.

Misc:

distribute – Supported python-3 since v0.6.2 (released 2009-09-26).

Django – The FAQ indicates they believe porting to python-3 will require dropping support for older python-2.x versions. According to mailing lists posts I could find, they intend to support the main python version that is used by a currently supported RHEL, so it could take quite some time before they do this. However, an initial porting effort showed that quite some progress could be made without having to drop support for current python-2.x versions.

PIL – The release notes for 1.1.7 states that “A version of 1.1.7 for 3.X will be released later”, but that was posted on 2009-11-15 and there is a lack of further updates on its status.

pygame – Some python-3 support was available in 1.9.0 (released 2009-07-31) and the number of modules that work with python-3 looks set to markedly increase with 1.9.2 (due to numpy now being python-3 compatible). Its successor (pgreloaded) similarly supports python-3.

Twisted – There is a python-3.x milestone in their bug tracker, but there appears to be no dedicated porting effort. Having a quick look at its dependencies, there may need to be some downstream work before it is started anyway. Also, their buildbot that uses the “-3″ warnings flag shows there is a lot of groundwork to be done.

So depending on exactly what you want to do, python-3.x might be an option. Graphical application programming, scientific programming and gaming all have reasonable support currently and that only looks set to get better in the near future. However, if you want to do anything web based or interact with a database other than MySQL, it looks like python-3 support could be quite some time away.

Posted in Software on December 4th, 2010 by Allan – Comments Off

When Good Features Go Bad!

Sometimes people have really good ideas of features that should be included in software that end up having unintended side effects. This is one of those stories.

Enter Liferea, probably the best RSS feed reader written in GTK2. You can tell it is good because they provide links to alternative feed readers on their front page… that shows confidence!

But it has one “feature” that has had me tearing my hair out. Whenever it checks a feed and encounters a redirect, it blindly updates the feeds URL. While that seems a good idea (as it prevents constant redirects), it causes some issues… Imagine opening your feed reader in a hotel with a paywall or terms of service page where you have to agree to go any further and all your feeds get their URL changed.

When that happened to me, I grabbed the source code and deleted the entire URL update segment. Given the Arch policies on patching and upstream issues, this is the sort of “fix” that would not make it into our repos so I have had to maintain my own Liferea package for a few months now. But that is all to change with a patch being accepted upstream. Of course, the patch does things the correct way and updates URLs that redirect only with HTTP 301 (Moved Permanently) codes.

So now I just have to trust that hotels do not issue a 301 codes to their paywall. For some reason, I am still worried…

Posted in Software on November 19th, 2010 by Allan – 1 Comment

Recovering Files From an NTFS Drive

Important lessons for computers owners:

  1. Set up a backup system
  2. Use that backup system regularly

It turns out that while I set up a backup system for my wife’s computer, the last time it was ran was sometime late 2009… So when the primary Windows partition on that computer no longer booted; was not recognised as a Windows install by the Windows install CD rescue option; and would not mount using the ntfs-3g driver from Linux, I knew I was in trouble! Luckily, I had split the Windows install to have a separate partition where most of her data (photos, music, documents) were sitting and that partition was still mountable. So, in the worst case scenario, the total damage would be limited to Firefox and Thunderbird profiles and whatever was saved on the desktop.

Recovering more data from the hard-drive was not particularly difficult as the hard-drive still basically worked. In that case, the first step in any data recovery attempt is to create a disk image of the broken partition. This prevents any further damage to the hard-drive making you life worse. A good tool for the job is GNU ddrescure (which may be named ddrescue or gddrescue depending on your Linux distribution). Make sure you have the GNU version as the other software with the same name is less user friendly. It can take a long time to take an image of a hard-drive so having some nice output and being able to stop and restart at any stage are quite essential. Creating an image is as simple as running (as root):

ddrescue -r3 /dev/sda2 imagefile logfile
The log file shows that the hard-drive had quite a few corrupt parts, all relatively small and close together so my guess is some sort of physical damage preventing those being read.

# Rescue Logfile. Created by GNU ddrescue version 1.13
# Command line: ddrescue -r3 /dev/sda2 imagefile logfile
# current_pos current_status
0x30D31E00 +
# pos size status
0x00000000 0x309EB000 +
0x309EB000 0x00001000 -
0x309EC000 0x00073000 +
0x30A5F000 0x00002000 -
0x30A61000 0x00073000 +
0x30AD4000 0x00002000 -
0x30AD6000 0x00073000 +
0x30B49000 0x00002000 -
0x30B4B000 0x000FC000 +
0x30C47000 0x00001000 -
0x30C48000 0x00074000 +
0x30CBC000 0x00001000 -
0x30CBD000 0x00074000 +
0x30D31000 0x00001000 -
0x30D32000 0x7F5409E00 +

So now you have an image of your hard-drive, it is time to get those files out. Given the disk image was relatively complete, I went for software called Sleuth Kit. It has a frontend called Autopsy, which I found fairly useless apart from browsing the data. To use it, start Autopsy by pointing it at directory where all its files are to be stored and then point your web browser at the relevant place:

mkdir autopsydir
autopsy -d autopsydir
firefox http://localhost:9999/autopsy

While browsing the data was good to confirm that most of the files I wanted were still there, I just wanted to extract every single file possible from the image and I would then copy all the relevant stuff over to the new computer as necessary. Slueth Kit has some command-line tools for doing that; fls for listing files and icat for getting them. Using these, you can extract the files one at a time… but a simple script will automate extracting everything.

#!/bin/sh
IMAGE=../imagefile
fls -urp $IMAGE |
while read type inode name; do
   echo $name
   case $type in
      d/d) mkdir -p "$name" ;;
      r/r) [ ! -f "$name" ] && icat $IMAGE $(echo $inode | sed 's/://g') > "$name" ;;
   esac
done

That will take a long time to run, especially if it is your primary Windows install drive as it will have lots of small files to extract. So I made that script so that it can continue where it left off. Just do not stop one run in the middle of an important file or it will not finish extracting it. You might want to also touch hiberfil.sys and pagefile.sys in the extraction directory first as they are relatively useless and will take up a few gigabytes each.

So everything appears to have been recovered and I survived the lack of backup! Windows 7 has a good feature that reminds you to set up a backup and run it regularly, so hopefully I will not need to do this again.

Posted in Software on September 22nd, 2010 by Allan – Comments Off