technical |
TLDR: DEFLATE decompressor in 3K of RAM For a Pebble app I've been writing, I need to send images from the phone to
the watch and cache them in persistent storage on the watch. Since the
persistent storage is very limited (and the Bluetooth connection is relatively
slow) I need these to be as small as possible, and so my original plan was to
use the PNG format and
The constraintThe major constraint for Pebble watchapps is memory. On Pebble Classic apps
have 24K of RAM available for the compiled code ( Initially, trying to decompress something simply crashed the app. It took some
debug prints to determine that code in Huffman treesHuffman coding is a method to represent frequently used symbols with fewer bits. It uses a tree (otherwise referred to as a dictionary) to convert symbols to bits and vice versa. DEFLATE can use Huffman coding in two modes: dynamic and fixed. In dynamic mode, the compressor constructs an optimal tree based on the data being compressed. This results in the smallest representation of the actual input data; however, it has to include the computed tree in the output in order for a decompressor to know how to decode the data. In some cases the space used to serialise the tree negates the improvement in the input representation. In this case the compressor can used fixed mode, where it uses a static tree defined by the DEFLATE spec. Since the decompressor knows what this static tree is, it doesn't need to be serialised in the output. The original tinf implementation builds this fixed tree in The dynamic trees are themselves serialised using Huffman encoding (yo dawg).
The resultWith the stack saving I was able to move the heap allocation back to the stack.
(Since the stack memory can't be used for anything else it's kind of free
because it allows the non-stack memory to be used for something else.) The end
result is 1.2K of |
|||
I recently bought a Raspberry Pi, which is a credit card sized computer with an ARM processor. I'm using it as my TV frontend, running Raspbian and XBMC. I'm building my own packages for XBMC since it requires the latest development version. I initially installed my Pi with the foundation image, but found that it included a lot of packages which I didn't need. Since I have a slight obsession about doing things as efficiently as possible, I decided to build my own image with XBMC from scratch. I implemented a script in Bash, mkraspbianxbmc.sh which does this. It uses debootstrap to install a minimal Raspbian system in a chroot. It then installs XBMC and a couple extra packages, and does some necessary configuration. Finally it creates an image file with the necessary partitions, creates the filesystems, and copies the installation into the image file. The resultant image fits onto a 1GiB SD card. You can download a pre-built image from this page. The script can be modified to build images with different packages, or even a very minimal image which fits onto a 512MiB SD card. |
|||
Extracting specific track segments from a trackI have an i-Blue 747A+ GPS logger which I use to track my runs (amongst other things). Afterwards I use BT747 to retrieve the data from the device and create a GPX file of the run, which I then upload to Endomondo which gives me nice graphs and statistics. I need to modify the GPX file slightly before I can do so however: I use the button on the device to mark the beginning and end of the run, which appear as waypoints in the GPX file. BT747 creates separate track segments (within a single track) between each waypoint, but Endomondo ignores these. I therefore need to extract the single track segment covering the actual run and create a new GPX file with just that segment. I therefore wrote a script in Python, splittrack.py, to do this. It uses the gpxdata library to parse the input file, locates any track segment which match a set of time, distance, displacement and speed criteria1, and outputs a new GPX file with just those.
Integrating heart rate dataI then recently bought an Oregon Scientific WM100 heart rate logger. It listens to the broadcasts from a heart rate strap2 and records the measurements every 2 seconds. I retrieve the data using the wm100 driver for Linux which writes a CSV file like this:
In order to get this data into Endomondo, I needed to combine the GPS trace with the HRM data into a single file format which Endomondo accepts. I initially started implementing a library for the TCX format3, but then discovered that there is a GPX extension for including heart rata data which Endomondo accepts. So I wrote a script in Python, wm100gpx.py, which reads the input GPX and CSV files, merges the heart rate measurements into the GPX records, and outputs a new GPX file.
The entries look like this: <trkpt lat="37.392051" lon="-122.090240"> <ele>-44.400761</ele> <time>2012-10-15T01:20:13Z</time> <extensions> <gpxtpx:TrackPointExtension> <gpxtpx:hr>175</gpxtpx:hr> </gpxtpx:TrackPointExtension> </extensions> </trkpt>
|
|||
I recently got a Raspberry Pi, which is an ARM based device which runs Linux. My goal is to use this as my HTPC running XBMC, so that I can move the fileserver out the lounge. Edit: I've moved the latest information and updates about these packages to this page. BuildingI installed Raspbian on my RPi, which is basically a rebuild of Debian Wheezy specifically for the RPi (targeting the ARMv6 architecture with hard float). I found various instructions on how to build XBMC for Raspbian, but none of them were in the form of deb packages, and installing software without packages just makes me queezy. So I went off and built it myself. Since the RPi is relatively low powered, I built the package on my laptop using qemu-user, which emulates binaries with a different architecture. I based the packaging on the XBMC package in Wheezy, and the source is from the xbmc-rbp branch on GitHub. I made the modifications to the source as per this forum post and added an initscript so that it can automatically start at bootup. InstallingThe easiest way to install the package is to add my archive to your system. To do this, store the
following in
and then import the archive signing key:
You can then install it as you would with any other package, for example, with
(If you don't want to configure my archive you can download the packages manually, but you'll have
to deal with all the dependencies. Note that it requires a newer The user which you're going to run XBMC as needs to be a member of the following groups:
RunningTo run XBMC, run If you want XBMC to automatically start when the system boots, edit
You also need to set the user which XBMC should run as (the ConfigurationThe following settings in advancedsettings.xml decreases the CPU usage while showing the UI. Disabling the RSS feeds also helps with this.
RebuildingIf you want to rebuild this package with a different source (e.g. a later Git revision), you need to
prepare the source by running |
|||
My wife recently got a Samsung Exhibit II 4G Android phone to replace her aging Nokia E63. Migrating her contacts was accomplished fairly easily by exporting them to CSV with Nokia PC Suite and then importing them into Google Contacts. Migrating SMSes was not so trivial however. Other approachesThere are a couple methods floating around the web, but none were suitable. This one uses Gammu to retrieve the SMSes from the Nokia, and then a script to convert them to an XML format readable by SMS Backup & Restore. It turns out that Gammu doesn't work on Symbian S60v3 devices however. This script can convert SMSes exported by the MsgExport app to XML for SMS Backup & Restore, but I didn't feel like paying for it or dealing with the Ovi Store. VeryAndroid is a Windows application which can convert SMSes from Nokia PC Suite CSV format and sync them directly to an Android device, and Nokia2AndroidSMS can convert SMSes from the OVI Suite database to XML for SMS Backup & Restore. I didn't want to deal with more Windows software though, so I just decided to write my own. FormatsI already had the Nokia PC Suite installed and was using it to migrate contacts, so I decided to work with the CSV output it generates for messages. A received SMS looks like this:
and a sent SMS looks like this:
The fields are:
Fields 4 and 6 are always empty for SMSes (they are probably used for MMSes, one being the message subject). I also decided to generate XML for the SMS Backup & Restore app. The XML format looks like this:
but can be reduced down to this:
The attributes of the
The scriptI implemented a script called [nokia2android.py] in [Python] to convert one or more CSV files to this XML format.
The XML file can then be transferred to the Android device (using USB or Bluetooth) and stored in
|
|||
Apple trackpads don't have separate buttons, the entire trackpad is itself a clickable button. The default OS X behaviour is to treat clicking the pad with two fingers as the right button, and clicking the pad with three fingers as the middle button. Enabling tap to click (i.e. touching the trackpad but not actually clicking it) tends to result in false positives since the trackpad is so big. I therefore setup this behaviour under Ubuntu Oneiric. When I upgraded to Ubuntu Precise two finger clicks started registering as the left button. (Three finger clicks still worked.) It turns out that this is due to the new clickpad support in Precise. The solution is to disable the
|
|||
I recorded some presentations at work recently using a Canon Vixia HF R21 video camera, and needed to encode the videos to lower resolution. It took me longer to figure this out than I expected, so I thought that I'd document the details. InputThe camera exposes an MTP interface when plugged in using USB, and the video
files are stored under EncodingThe best codecs in terms of storage space efficient seem to be H.264 and AAC, so that's what I chose to encode the output in. I wanted a good quality version at 720p, and a lower quality version at 480p. Since these were presentations, single channel audio is sufficient. I tried using both libav (previously ffmpeg) and mencoder and had trouble with both, but eventually got libav working. Good qualityThe command I used for good quality was:
Audio is reduced to mono and encoded in AAC at 128kbps. Video is resized to
1280x720, deinterlaced, and encoded in H.264 with the default quality
settings. To adjust the quality one can use the On my Intel Core2 Duo P8600 this encoded at 10fps with a video bitrate of
700kbps and audio bitrate of 83kbps. This puts an hour of video at 339MiB.
Adding Low qualityThe command I used for low quality was:
Audio is encoded the same as above. Video is resized to 848x480, deinterlaced and encoded in H.264. Apparently codecs are more efficient when the resolutions are a multiple of 16 which is why the video is resized to 854 and then reduced to 848. This encoded at 20fps with a video bitrate of 300kbps and audio bitrate of 83kbps. This puts an hour of video at 165MiB.
|
|||
After over a year of development, we have finally released Ibid. Ibid is a general purpose chat bot written in Python. We've suffered from a bit of feature creep, so despite being a 0.1 release it can talk 7 messaging protocols and has over 100 features provided by plugins. I think we also have an excellent architecture and very developer friendly plugin API. The 0.1.0 release can be downloaded from Launchpad or installed from our PPA. |
|||
SuperGenPass and Password Composer are password generators, which generate different passwords for each site you use based on a single master password. This gives you the convenience of only remembering one password as well as the security of using different (and strong) passwords for each site. This means that you won't have all your accounts compromised when1 one of them is compromised. Most password generators are implemented as browser extensions or bookmarklets, since they are most frequently needed in a web browser. I've been wanting to start using a password generator, but I wanted to be sure that I could access my accounts even if I didn't have a web browser accessible. The two situations I could think of were a command line only system (e.g. SSH) and my cellphone2. Surprisingly, I couldn't find a command line implementation of SuperGenPass, so I wrote one in Python. I also couldn't find any J2ME or Symbian implementations, and so wrote my own one in J2ME. They both support subdomain stripping and configurable password lengths. They don't support salted passwords. I chose SuperGenPass over Password Composer because it uses a better scheme. Password Composer only uses hex characters, whereas SuperGenPass uses a base64 encoded hash. SuperGenPass also hashes the password multiple times (which would slow down a brute force attack to find the master password) and imposes complexity requirements on the generated password (which reduces the chances that the generated password can be brute forced). |
|||
Following on from yesterday's post, I decided to try implement proper content negotiation. After a fair amount of time spent getting to grips with [Lua][], I got a [script][] which works very nicely. It implements [server driven][] content negotiation for [media types][mime]. The basic idea of content negotiation is that a resource (e.g., this [graph][])
exists in multiple formats (in this case, [SVG][graph-svg], [PNG][graph-png] and
[GIF][graph-gif]). When a user agent requests the resource, it indicates which
formats it understands by listing them in the (The following description assumes knowledge of the The script works by searching the directory for files with the requested name
but with an additional extension (each of which is a variant). The [media
type][mime] is inferred from the extension using Some browsers include wildcard entries such as To install the [script][], download and save it somewhere (such as
|
|||
