linux

Building Raspbian images for Raspberry Pi

I recently bought a Raspberry Pi, which is a credit card sized computer with an ARM processor. I'm using it as my TV frontend, running Raspbian and XBMC. I'm building my own packages for XBMC since it requires the latest development version.

I initially installed my Pi with the foundation image, but found that it included a lot of packages which I didn't need. Since I have a slight obsession about doing things as efficiently as possible, I decided to build my own image with XBMC from scratch.

I implemented a script in Bash, mkraspbianxbmc.sh which does this. It uses debootstrap to install a minimal Raspbian system in a chroot. It then installs XBMC and a couple extra packages, and does some necessary configuration. Finally it creates an image file with the necessary partitions, creates the filesystems, and copies the installation into the image file. The resultant image fits onto a 1GiB SD card. You can download a pre-built image from this page.

The script can be modified to build images with different packages, or even a very minimal image which fits onto a 512MiB SD card.

Working with GPX files in Python

Extracting specific track segments from a track

I have an i-Blue 747A+ GPS logger which I use to track my runs (amongst other things). Afterwards I use BT747 to retrieve the data from the device and create a GPX file of the run, which I then upload to Endomondo which gives me nice graphs and statistics.

I need to modify the GPX file slightly before I can do so however: I use the button on the device to mark the beginning and end of the run, which appear as waypoints in the GPX file. BT747 creates separate track segments (within a single track) between each waypoint, but Endomondo ignores these. I therefore need to extract the single track segment covering the actual run and create a new GPX file with just that segment. I therefore wrote a script in Python, splittrack.py, to do this. It uses the gpxdata library to parse the input file, locates any track segment which match a set of time, distance, displacement and speed criteria1, and outputs a new GPX file with just those.

% splittrack.py mgorven-20121015_0109.gpx > run-20121014.gpx
Reading mgorven-20121015_0109.gpx
<TrackSegment (23 points)> covers 21m over 0:00:22 at 1.0m/s average speed with 4m displacement
<TrackSegment (904 points)> covers 3018m over 0:15:03 at 3.3m/s average speed with 8m displacement
Adding <TrackSegment (904 points)>
<TrackSegment (4 points)> covers 3m over 0:00:03 at 1.3m/s average speed with 3m displacement

Integrating heart rate data

I then recently bought an Oregon Scientific WM100 heart rate logger. It listens to the broadcasts from a heart rate strap2 and records the measurements every 2 seconds. I retrieve the data using the wm100 driver for Linux which writes a CSV file like this:

Name,2012-10-14T18:08:27
Description,
Date,10/14/2012
Time,18:08:27
SamplingRate,2
HeartRate
,80
,78
,76
,75

In order to get this data into Endomondo, I needed to combine the GPS trace with the HRM data into a single file format which Endomondo accepts. I initially started implementing a library for the TCX format3, but then discovered that there is a GPX extension for including heart rata data which Endomondo accepts. So I wrote a script in Python, wm100gpx.py, which reads the input GPX and CSV files, merges the heart rate measurements into the GPX records, and outputs a new GPX file.

% wm100gpx.py 2012-10-14T18:08:27.csv < mgorven-20121015_0109.gpx > run-20121014.gpx

The entries look like this:

<trkpt lat="37.392051" lon="-122.090240">
  <ele>-44.400761</ele>
  <time>2012-10-15T01:20:13Z</time>
  <extensions>
     <gpxtpx:TrackPointExtension>
         <gpxtpx:hr>175</gpxtpx:hr>
     </gpxtpx:TrackPointExtension>
    </extensions>
</trkpt>


  1. I actually initially wrote this to find tracklogs of runs amongst all my tracklogs. 

  2. I use a strap from an entry level Nike triax C3 heart rate monitor watch. 

  3. Which is quite exhaustive... 

XBMC packages for Raspberry Pi running Raspbian

I recently got a Raspberry Pi, which is an ARM based device which runs Linux. My goal is to use this as my HTPC running XBMC, so that I can move the fileserver out the lounge.

Edit: I've moved the latest information and updates about these packages to this page.

Building

I installed Raspbian on my RPi, which is basically a rebuild of Debian Wheezy specifically for the RPi (targeting the ARMv6 architecture with hard float). I found various instructions on how to build XBMC for Raspbian, but none of them were in the form of deb packages, and installing software without packages just makes me queezy. So I went off and built it myself.

Since the RPi is relatively low powered, I built the package on my laptop using qemu-user, which emulates binaries with a different architecture. I based the packaging on the XBMC package in Wheezy, and the source is from the xbmc-rbp branch on GitHub. I made the modifications to the source as per this forum post and added an initscript so that it can automatically start at bootup.

Installing

The easiest way to install the package is to add my archive to your system. To do this, store the following in /etc/apt/sources.list.d/mene.list:

deb http://archive.mene.za.net/raspbian wheezy contrib

and then import the archive signing key:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-key 5243CDED

You can then install it as you would with any other package, for example, with apt-get:

sudo apt-get install xbmc

(If you don't want to configure my archive you can download the packages manually, but you'll have to deal with all the dependencies. Note that it requires a newer libcec package which is also in my archive.)

The user which you're going to run XBMC as needs to be a member of the following groups:

audio video input dialout plugdev

Running

To run XBMC, run xbmc-standalone from a VT (i.e. not under X). XBMC accesses the display directly and not via Xorg.

If you want XBMC to automatically start when the system boots, edit /etc/default/xbmc and change ENABLED to 1:

ENABLED=1

You also need to set the user which XBMC should run as (the xbmc user is not automatically created at the moment). Run sudo service xbmc start to test this.

Configuration

The following settings in advancedsettings.xml decreases the CPU usage while showing the UI. Disabling the RSS feeds also helps with this.

<advancedsettings>
    <gui>
        <algorithmdirtyregions>3</algorithmdirtyregions>
        <nofliptimeout>0</nofliptimeout>
    </gui>
</advancedsettings>

Rebuilding

If you want to rebuild this package with a different source (e.g. a later Git revision), you need to prepare the source by running ./bootstrap before creating the orig.tar.gz. You obviously need all the build dependencies (which should be listed in the packaging), as well as the VideoCoreIV libraries in /opt. You probably want to set DEB_BUILD_OPTIONS=nocheck parallel=4 to disable the tests (they failed to build for me), and speed up the build by using more than one core. You can find the packaging in my archive or my Bazaar repository.

Clicking on Apple Macbook Pro trackpads under Ubuntu Precise

Apple trackpads don't have separate buttons, the entire trackpad is itself a clickable button. The default OS X behaviour is to treat clicking the pad with two fingers as the right button, and clicking the pad with three fingers as the middle button. Enabling tap to click (i.e. touching the trackpad but not actually clicking it) tends to result in false positives since the trackpad is so big. I therefore setup this behaviour under Ubuntu Oneiric.

When I upgraded to Ubuntu Precise two finger clicks started registering as the left button. (Three finger clicks still worked.) It turns out that this is due to the new clickpad support in Precise. The solution is to disable the ClickPad attribute. My Synaptics configuration now looks like this:

TapButton1              = 0
TapButton2              = 0
TapButton3              = 0
ClickFinger1            = 1
ClickFinger2            = 3
ClickFinger3            = 2
ClickPad                = 0

Sharing links from Konqueror, including to IRC

I follow the main feeds of a couple social news sites (namely Digg, Reddit and Muti). When I find an article which I like, I go back and vote it up on the site. However, when I come across good articles via other sources, I don't submit them to these news sites (or try to find out if they've already been submitted) simply because it's too much effort.

When I started aggregating my activity on these sites on my blog and on FriendFeed, I needed a way to share pages that I didn't get to via one of these social news sites. I ended up setting up Delicious because I found a plugin for Konqueror which made it easy to bookmark pages.

I still wanted to solve the original problem though, and so started looking for an easy way to submit links to these sites from Konqueror. Konqueror has a feature called service menus which allows you to add entries to the context menu of files. I then needed to work out how to submit links to these services, which turned out to simply involve loading a URL with a query parameter specifying the link you want to share.

I created entries for Reddit, Digg, Muti, Delicious, Facebook and Google Bookmarks. These take you to the submission page of the service where you can fill in the title1. Digg and Reddit will show existing submissions if the link has already been submitted.

I often share links on IRC, and wondered if I could integrate that with my menu. It turns out that WeeChat has a control socket, and I could send messages by piping them to the socket. I therefore wrote a script which prompted me for a headline or excerpt using kdialog, and then sent the link to the specified channel. My menu now looks like this:

sharemenu.png

If you want to set this up yourself, download share.desktop and put it in ~/.kde/share/apps/konqueror/servicemenus. If you want the icons, download shareicons.tar.gz, extract them somewhere, and fix the paths in social.desktop2. To setup the IRC feature (assuming you're using WeeChat), download postirc.sh and save it in ~/bin/. You will need to change the commands in social.desktop depending on the servers and channels you wish to use.


  1. One shortcoming is that the title of the page is not automatically filled in. 

  2. I couldn't work out how to use relative paths, or ~. 

Windows users finally get circular scrolling

When I first started using Linux four years ago, one of the most useful features I discovered was circular scrolling on touchpads. (For those that don't know, this allows you to scroll up and down by moving your finger in a circle.) Traditional scrolling now feels very clumsy, and I find it awkward when using a laptop which doesn't have this feature (such as those running Windows). According to the changelog for the XOrg/XFree86 Synaptics driver, this feature was added in February 2004.

I happened to come across the news today that Synaptics have added a feature called ChiralTouch Technology to the latest version of their Windows drivers. This so-called "technology" provides "the ability to scroll continuously with a circular motion." This basically means that they have finally gotten round to implementing a very useful feature which Linux users have had for over four years.

In some respects proprietary software is way behind FOSS in terms of features and usability, and this example also shows how proprietary software uses ideas which were first implemented in FOSS.

Vim syntax highlighting for irssi IRC logs

When I occasionally read [IRC][] logs saved by [irssi][], I find the lack of colouring rather annoying and find that I can't read them very quickly. I finally got round to writing a syntax highlighting plugin for [Vim][] in order to correct this. The colours could probably do with some improvement, but it's much better than before.

In case anyone else finds this useful, I have attached the plugin to this post. To use it, save [irssilog.vim][irssilog.vim] in ~/.vim/syntax/ and enter the following command to use it with the current file.

:set syntax=irssilog

If you want Vim to automatically detect the file type, add the following to ~/.vim/ftdetect/irssilog.vim1.

au BufRead,BufNewFile */irclogs*.log    set filetype=irssilog

  1. This relies on the logs being stored in the default location of ~/irclogs/

My personal backup solution

I've been using an external harddrive to store backups of my laptop for a while now. At first I manually created a set of compressed tar archives about once a month. That was a bad system though because it used a lot of space and was a mission to retrieve files from backups. I then started using pdumpfs, which can do incremental backups by hard linking files which haven't changed. The problem I found with it however was that if a file's ownership or timestamps changed it wouldn't be hard linked even if the content hadn't changed.

I therefore set out to find a better backup solution. My requirements were as follows.

  1. Incremental backups
  2. Easy to access specific files from backups
  3. Able to delete certain backups, preferably arbitrarily1
  4. Compression
  5. Encryption

I finally settled on storeBackup which supports everything except number 5. It works similarly to pdumpfs, except it stores ownership and timestamp data separately and therefore can still hard link identical files even if these change. It compresses on a per file basis, which makes it easy to access specific files (as opposed to having to find them in an archive). Old backups can be deleted arbitrarily since they are only related by hard links. I then added encryption by backing up to an encfs encrypted directory.


  1. I want to be able to backup every week, but then delete old backups so that I have one backup per month for the last year. 

Routing by port number

Due to a very restrictive firewall at the CHPC, I need to run a VPN to get access to things like email, Jabber and SSH. This however degrades my web browsing experience, since that gets tunnelled as well. I therefore wanted a setup where only ports which are blocked get tunnelled through the VPN, while everything else goes out normally.

The routing part was fairly straightforward, which consists of an iptables rule to mark certain packets, and an alternate routing table for these marked packets. I first created a name for the new table by adding the following to /etc/iproute2/rt_tables.

10  vpn

I then added a default route to the new table specifying the IP address of the VPN server and the VPN interface, and a rule to use this table for packets marked by iptables.

ip route add default via 10.8.0.3 dev tun0 table vpn
ip rule add fwmark 0x1 table vpn

The following iptables rule will mark packets destined to the listed port numbers. Note that this is for packets originating from the firewall host — if you want this to apply to packets forwarded for other hosts it must be in the PREROUTING chain.

iptables -t mangle -A OUTPUT -p tcp -m multiport --dports 22,995,587,5223 -j MARK --set-mark 0x1

The actual routing worked, but packets were being sent with the wrong source IP. I therefore needed to NAT packets going out on the VPN interface (the IP address is the local IP of the VPN connection).

iptables -t nat -A POSTROUTING -o tun0 -j SNAT --to 10.8.0.4

I could then see packets going out on the VPN interface with the correct source IP as well as the replies, but it still wasn't working. I eventually discovered that rp_filter must be disabled in order for this to work.

echo 0 > /proc/sys/net/ipv4/conf/tun0/rp_filter

Cisco un-Clean Access

The [CHPC][] installed a new network this past weekend as part of the [SANReN][] project. The new network consists of [Cisco][] equipment, including their [NAC][] (or "Clean Access") system. This requires all clients to authenticate before they are allowed access to the network, and can also enforce a configured security policy (such as requiring operating system updates and anti-virus).

The system works as follows. By default, the ports on the switch are in an "unauthenticated" [VLAN][]. When a client is connected, it is provided with an IP address (via [DHCP][]) in an "unauthenticated" subnet. The system then presents a captive portal which requires the user to authenticate with a username and password using their browser. If the authentication is successful, the port is moved to a different VLAN (depending on the user's access level), and the switch briefly disconnects the link which causes the client to negotiate a new IP address (in a different subnet).

Before the portal presents the login page it requires that a [Java applet][] be run on the client. The applet gathers various bits of information about the client (including the operating system) and submits this information to the portal. (I assume that the portal uses this information to determine what policies must be enforced. In our setup, Windows machines must have the Clean Access Client installed, while Linux and Mac OS X machines are simply allowed access.) The portal then presents the login page.

Being a geek, I wasn't very happy to go through this rigmarole everytime I connected to the network. (I also couldn't use my [normal browser][konq] since the applet didn't work in it.) So I set out to automate the process. Initially I tried to script everything (including the Java applet) but then I noticed that the output of the applet wasn't sent with the login form submission. The only other information the form contained was a session key and random string, both of which were present on the [HTML][] page which contained the applet. A manual test confirmed that the login page could be submitted successfully as long as the session key and random string were correct — the applet could be bypassed.

I quickly scripted the login process using a

[] script and [wget][]. I then installed it in <code>/etc/network/if-up.d</code> after adding some logic to only execute if the current IP address was on the unauthenticated network. The result is that I can plug in the cable, and my machine automatically authenticates to the system.
 
While searching for information about the Clean Access system, I came across this [Slashdot article][] about a guy who was suspended from university for bypassing the Clean Access checks. I only realised last night that this is exactly what my script does![^1] I haven't tested it on Windows yet, but the only possible change I can think of is to change the [user agent][]. Seriously Cisco, the fact that I managed to bypass the applet simply by submitting the login form programmatically is ridiculous.
 
I have attached my script to this post. The way in which I have parsed the HTML page is rather ugly and likely to only work on this specific version of Clean Access. I plan to rewrite it in [Python][] sometime.
 
<strong>Update:</strong> I have rewritten the script in Python, which should be a bit more solid since it parses the HTML using a [DOM][]. The script requires [libxml2dom][] and [ipy][]. After configuring the parameters it can be dropped in <code>/etc/network/if-up.d</code>[^2] where it should run automatically.
 
[^1]: Note that it doesn't bypass the authentication: you still need a valid account in order to gain access.
 
[^2]: Make sure not to use a dot in the filename though.
 
<em>[CHPC]: Centre for High Performance Computing
</em>[SANReN]: South African National Research Network
<em>[NAC]: Network Admission Control
</em>[VLAN]: Virtual LAN
<em>[DHCP]: Dynamic Host Configuration Protocol
</em>[HTML]: HyperText Markup Language
*[DOM]: Document Object Model
 
[chpc]: http://www.chpc.ac.za/
[sanren]: http://www.meraka.org.za/sanren.htm
[cisco]: http://en.wikipedia.org/wiki/Cisco_Systems
[vlan]: http://en.wikipedia.org/wiki/Virtual_LAN
[dhcp]: http://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol
[nac]: http://en.wikipedia.org/wiki/Cisco_NAC_Appliance
[java applet]: http://en.wikipedia.org/wiki/Java_applet
[konq]: http://en.wikipedia.org/wiki/Konqueror
[Slashdot article]: http://it.slashdot.org/article.pl?sid=07/04/27/203232
[user agent]: http://en.wikipedia.org/wiki/User_agent
[python]: http://en.wikipedia.org/wiki/Python_(programming_language)
[html]: http://en.wikipedia.org/wiki/HTML
[wget]: http://en.wikipedia.org/wiki/Wget
[bash]: http://en.wikipedia.org/wiki/Bash
[dom]: http://en.wikipedia.org/wiki/Document_Object_Model
[libxml2dom]: http://www.boddie.org.uk/python/libxml2dom.html
[ipy]: http://russell.rucus.net/2008/ipy/

Syndicate content