technical

New Knab modules

2008-12-20T20:50:37+01:00

I've recently been doing a lot of hacking on Knab, which is the software behind the #clug IRC bot, Spinach. I've contributed a number of new modules, most of which are running on Spinach and are available from the main Bazaar repository.

Events

This module is basically a calendar feature which can store and retrieve events such as birthdays. It also handles recurring events (both with rules¹ or multiple dates).

<cocooncrash> Something happens on 21 December 2008 at 15:00
<Knab> yessir
<cocooncrash> When is something?
<Knab> something is on Sunday the 21st of December 2008 at 15:00:00
<cocooncrash> Forget event something
<Knab> I've forgotten something
 
<cocooncrash> My birthday happens on 22 March every year
<Knab> sure
<cocooncrash> How long until cocooncrash's birthday?
<Knab> your birthday is 13 weeks and 1 day away
<cocooncrash> How many days until my birthday?
<Knab> your birthday is 92 days away
 
<cocooncrash> When is Easter in 2010?
<Knab> Sun 4 Apr
<cocooncrash> What's happening on Tuesday
<Knab> National Chocolate Covered Anything Day and Day of Reconciliation
<cocooncrash> Search events for Outsider 
<Knab> Found: Be Friendly To Outsider Day, Be Hard On Outsider Day, Outsider's birthday, Outsider's dad's birthday,  (showing 4 of 4 results)

Twitter

This module retrieves messages posted on Twitter and identi.ca.

<cocooncrash> Last twitter update by iolproofreader
<Knab> "The ANC in the Eastern Cape is making it clear: it's either you're with us or not."
<cocooncrash> twitter 1067135354
<Knab> iolproofreader: ""Friend chases after golfer shot" http://tinyurl.com/4dn2lx"
<cocooncrash> http://twitter.com/iolproofreader/status/1067135354
<Knab> iolproofreader: "Police are planning on more patrols and more scrutiny for travellers crossing the border of Swaziland and Mozambique."

URLLengthen

This modules discovers the URL which a shortened URL redirects to.

<cocooncrash> http://is.gd/1D
<Knab> http://google.com/

URLShorten

This module creates a short URL using is.gd.

<cocooncrash> shorten http://google.com/
<Knab> http://is.gd/1D

HTTP

This module retrieves an HTTP URL and returns the result of the request.

<cocooncrash> get http://google.com
<Knab> 200 OK "Google"
<cocooncrash> Is google.com up?
<Knab> Yes, it's up
<cocooncrash> Is Twitter down?
<Knab> No, it's just you

GoogleDefine

This module gets definitions using Google.

<cocooncrash> gdefine spinach
<Knab> "dark green leaves; eaten cooked or raw in salads  ", "The Spinach (Eulithis mellinata) is a moth of the family Geometridae.

Subversion

This module retrieves commit messages from a Subversion repository.

<cocooncrash> last commit
<Knab> r721 by trb143 7 hours ago: "Update UI a bit to display columns"
<cocooncrash> commit 700 full
<Knab> r700 by mgorven on 2008/12/15 at 21:14:26 UTC: "Set non-existing config entries if requested with a default value, and create ~/.openlp when saving config file. " M /openlp-2/trunk/openlp/core/utils/linregistry.py, M /openlp-2/trunk/openlp/core/utils/confighelper.py

SummonJabber

This module summons people by sending them a message via Jabber.

<cocooncrash> cocooncrash.summon.jabber is cocooncrash@example.com
<Knab> gotcha
<cocooncrash> summon cocooncrash
<Knab> I've summoned cocooncrash via Jabber

Feeds

This module retrieves RSS and Atom feeds.

<cocooncrash> Add feed http://www.news24.com/news24RSSFeed/0,,2,00.xml as News24
<Knab> okay
<cocooncrash> latest articles from news24
<Knab> 0: "SA's first San party launches", 1: "Pakistan mall collapse: 3 dead", 2: "Iran warship to fight pirates", 3: "Westwood storms into lead", 4: "US Embassy given the boot", 5: "Jennifer Aniston's chilli Xmas"
<cocooncrash> article 0 from News24
<Knab> "SA's first San party launches" http://www.news24.com/News24/South_Africa/Politics/0,,2-7-12_2444493,00.... : The first San political party has been launched at Upington in the Northern Cape. 
<cocooncrash> article /Palin/ from news24
<Knab> "Drama in Palin household" http://www.news24.com/News24/World/News/0,,2-10-1462_2444295,00.html : The mother of  an 18-year-old man who plans to marry Alaska Governor Sarah Palin's  pregnant daughter, Bristol, has been arrested on drug charges.
<cocooncrash> list feeds
<Knab> m&g, clug park, haiybo, TechCrunch, onion, News24
<cocooncrash> remove news24 feed
<Knab> okay

This feature is provided by the Date::Manip library. ↩

Playing with Python and IRC

2008-10-31T23:58:40+01:00

I wrote three IRC bots in Python this last week (although one was a rewrite). They probably aren't very useful to most people, but I'm going to share them anyway in case someone finds them interesting.

The first one was prompted by Adrian, who is maintaining a countdown until his wedding as a factoid in Spinach. Since Knab doesn't actually support countdowns, it has to be updated manually. This clearly isn't the Right Way to do this, and so I hacked together a script which connects to IRC and teaches Spinach the updated factoid. I run this as a daily cronjob to keep the countdown up to date.

As is usually the case with Python, there was already a library for accessing IRC, namely irclib. It isn't documented very well, but has a couple example scripts which are fairly easy to follow. It follows an event based model, so you write functions which will be called when certain events occur (such as receiving a message).

The final of the Currie Cup was held on Saturday (which my team (the Sharks) won), and I followed the match online using SuperSport's live score site¹. I then thought that it would be cool to have the score announced on IRC when it changed, and since I was bored I wrote a simple bot to do this. It worked well, but was very simple in that it only supported one hardcoded channel and one hardcoded game.

Since I was also bored on Sunday I rewrote this bot properly. I added a subscription mechanism so that channels and users can subscribe and unsubscribe to games by sending the bot a command. It's mostly working except for listing the available games (since there aren't any rugby games coming up which means that I can't test it ;-) ). Games are specified by the ID used by SuperSport's site, and finding the right ID is currently a manual process.

I'm not really a sports fan — I just enjoy bragging when we do win ;-) ↩

Sharing links from Konqueror, including to IRC

2008-10-24T17:09:12+02:00

I follow the main feeds of a couple social news sites (namely Digg, Reddit and Muti). When I find an article which I like, I go back and vote it up on the site. However, when I come across good articles via other sources, I don't submit them to these news sites (or try to find out if they've already been submitted) simply because it's too much effort.

When I started aggregating my activity on these sites on my blog and on FriendFeed, I needed a way to share pages that I didn't get to via one of these social news sites. I ended up setting up Delicious because I found a plugin for Konqueror which made it easy to bookmark pages.

I still wanted to solve the original problem though, and so started looking for an easy way to submit links to these sites from Konqueror. Konqueror has a feature called service menus which allows you to add entries to the context menu of files. I then needed to work out how to submit links to these services, which turned out to simply involve loading a URL with a query parameter specifying the link you want to share.

I created entries for Reddit, Digg, Muti, Delicious, Facebook and Google Bookmarks. These take you to the submission page of the service where you can fill in the title¹. Digg and Reddit will show existing submissions if the link has already been submitted.

I often share links on IRC, and wondered if I could integrate that with my menu. It turns out that WeeChat has a control socket, and I could send messages by piping them to the socket. I therefore wrote a script which prompted me for a headline or excerpt using kdialog, and then sent the link to the specified channel. My menu now looks like this:

If you want to set this up yourself, download share.desktop and put it in ~/.kde/share/apps/konqueror/servicemenus. If you want the icons, download shareicons.tar.gz, extract them somewhere, and fix the paths in social.desktop². To setup the IRC feature (assuming you're using WeeChat), download postirc.sh and save it in ~/bin/. You will need to change the commands in social.desktop depending on the servers and channels you wish to use.

One shortcoming is that the title of the page is not automatically filled in. ↩
I couldn't work out how to use relative paths, or ~. ↩

Windows users finally get circular scrolling

2008-10-01T17:12:23+02:00

When I first started using Linux four years ago, one of the most useful features I discovered was circular scrolling on touchpads. (For those that don't know, this allows you to scroll up and down by moving your finger in a circle.) Traditional scrolling now feels very clumsy, and I find it awkward when using a laptop which doesn't have this feature (such as those running Windows). According to the changelog for the XOrg/XFree86 Synaptics driver, this feature was added in February 2004.

I happened to come across the news today that Synaptics have added a feature called ChiralTouch Technology to the latest version of their Windows drivers. This so-called "technology" provides "the ability to scroll continuously with a circular motion." This basically means that they have finally gotten round to implementing a very useful feature which Linux users have had for over four years.

In some respects proprietary software is way behind FOSS in terms of features and usability, and this example also shows how proprietary software uses ideas which were first implemented in FOSS.

Vim syntax highlighting for irssi IRC logs

2008-10-01T16:19:20+02:00

When I occasionally read IRC logs saved by irssi, I find the lack of colouring rather annoying and find that I can't read them very quickly. I finally got round to writing a syntax highlighting plugin for Vim in order to correct this. The colours could probably do with some improvement, but it's much better than before.

In case anyone else finds this useful, I have attached the plugin to this post. To use it, save irssilog.vim in ~/.vim/syntax/ and enter the following command to use it with the current file.

:set syntax=irssilog

If you want Vim to automatically detect the file type, add the following to ~/.vim/ftdetect/irssilog.vim¹.

au BufRead,BufNewFile /irclogs.log    set filetype=irssilog

This relies on the logs being stored in the default location of ~/irclogs/. ↩

My personal backup solution

2008-09-26T15:10:02+02:00

I've been using an external harddrive to store backups of my laptop for a while now. At first I manually created a set of compressed tar archives about once a month. That was a bad system though because it used a lot of space and was a mission to retrieve files from backups. I then started using pdumpfs, which can do incremental backups by hard linking files which haven't changed. The problem I found with it however was that if a file's ownership or timestamps changed it wouldn't be hard linked even if the content hadn't changed.

I therefore set out to find a better backup solution. My requirements were as follows.

Incremental backups
Easy to access specific files from backups
Able to delete certain backups, preferably arbitrarily¹
Compression
Encryption

I finally settled on storeBackup which supports everything except number 5. It works similarly to pdumpfs, except it stores ownership and timestamp data separately and therefore can still hard link identical files even if these change. It compresses on a per file basis, which makes it easy to access specific files (as opposed to having to find them in an archive). Old backups can be deleted arbitrarily since they are only related by hard links. I then added encryption by backing up to an encfs encrypted directory.

I want to be able to backup every week, but then delete old backups so that I have one backup per month for the last year. ↩

Downloading Google Talk logs

2008-08-17T12:01:14+02:00

I used Google Apps to host mail for this domain for a while, and wanted to close down the account since I don't use it anymore. Before I did that I wanted to move all the data onto my server. Transferring the emails was fairly straightforward using POP3, but I couldn't find a way to download the Google Talk logs. Gmail handles the logs as emails, but they aren't accessible using either POP3 or IMAP.

I therefore wrote a Python script which downloads the logs via the web interface. On Jeremy's suggestion I used BeautifulSoup to parse the HTML this time, which worked very well. The script works with both Google Apps and normal Gmail, although my account got locked twice while trying to download the 3500 logs in my account.

Routing by port number

2008-08-15T12:35:10+02:00

Due to a very restrictive firewall at the CHPC, I need to run a VPN to get access to things like email, Jabber and SSH. This however degrades my web browsing experience, since that gets tunnelled as well. I therefore wanted a setup where only ports which are blocked get tunnelled through the VPN, while everything else goes out normally.

The routing part was fairly straightforward, which consists of an iptables rule to mark certain packets, and an alternate routing table for these marked packets. I first created a name for the new table by adding the following to /etc/iproute2/rt_tables.

10  vpn

I then added a default route to the new table specifying the IP address of the VPN server and the VPN interface, and a rule to use this table for packets marked by iptables.

ip route add default via 10.8.0.3 dev tun0 table vpn
ip rule add fwmark 0x1 table vpn

The following iptables rule will mark packets destined to the listed port numbers. Note that this is for packets originating from the firewall host — if you want this to apply to packets forwarded for other hosts it must be in the PREROUTING chain.

iptables -t mangle -A OUTPUT -p tcp -m multiport --dports 22,995,587,5223 -j MARK --set-mark 0x1

The actual routing worked, but packets were being sent with the wrong source IP. I therefore needed to NAT packets going out on the VPN interface (the IP address is the local IP of the VPN connection).

iptables -t nat -A POSTROUTING -o tun0 -j SNAT --to 10.8.0.4

I could then see packets going out on the VPN interface with the correct source IP as well as the replies, but it still wasn't working. I eventually discovered that rp_filter must be disabled in order for this to work.

echo 0 > /proc/sys/net/ipv4/conf/tun0/rp_filter

Photo blogging via email with Drupal

2008-08-10T17:12:16+02:00

The one thing missing from my posting by email setup was support for images. The Mailsave module has finally been updated for Drupal 6, and so I can now submit attachments with email posts. The one shortcoming is that files are simply added to posts as normal attachments, and so images aren't automatically displayed. I therefore have to manually insert images in the body of the post, but I actually prefer this since it's a simpler system and gives me more control.

I also needed a way of resizing images on my phone since they are too big. I found the Nokia Image Editor¹ which seems to work fairly well, although it only allows resizing to specific resolutions.

It does work on my phone even though it's supposedly only for the Nokia 3250. ↩

Cisco un-Clean Access

2008-08-06T11:09:36+02:00

The CHPC installed a new network this past weekend as part of the SANReN project. The new network consists of Cisco equipment, including their NAC (or "Clean Access") system. This requires all clients to authenticate before they are allowed access to the network, and can also enforce a configured security policy (such as requiring operating system updates and anti-virus).

The system works as follows. By default, the ports on the switch are in an "unauthenticated" VLAN. When a client is connected, it is provided with an IP address (via DHCP) in an "unauthenticated" subnet. The system then presents a captive portal which requires the user to authenticate with a username and password using their browser. If the authentication is successful, the port is moved to a different VLAN (depending on the user's access level), and the switch briefly disconnects the link which causes the client to negotiate a new IP address (in a different subnet).

Before the portal presents the login page it requires that a Java applet be run on the client. The applet gathers various bits of information about the client (including the operating system) and submits this information to the portal. (I assume that the portal uses this information to determine what policies must be enforced. In our setup, Windows machines must have the Clean Access Client installed, while Linux and Mac OS X machines are simply allowed access.) The portal then presents the login page.

Being a geek, I wasn't very happy to go through this rigmarole everytime I connected to the network. (I also couldn't use my normal browser since the applet didn't work in it.) So I set out to automate the process. Initially I tried to script everything (including the Java applet) but then I noticed that the output of the applet wasn't sent with the login form submission. The only other information the form contained was a session key and random string, both of which were present on the HTML page which contained the applet. A manual test confirmed that the login page could be submitted successfully as long as the session key and random string were correct — the applet could be bypassed.

I quickly scripted the login process using a bash script and wget. I then installed it in /etc/network/if-up.d after adding some logic to only execute if the current IP address was on the unauthenticated network. The result is that I can plug in the cable, and my machine automatically authenticates to the system.

While searching for information about the Clean Access system, I came across this Slashdot article about a guy who was suspended from university for bypassing the Clean Access checks. I only realised last night that this is exactly what my script does!¹ I haven't tested it on Windows yet, but the only possible change I can think of is to change the user agent. Seriously Cisco, the fact that I managed to bypass the applet simply by submitting the login form programmatically is ridiculous.

I have attached my script to this post. The way in which I have parsed the HTML page is rather ugly and likely to only work on this specific version of Clean Access. I plan to rewrite it in Python sometime.

Update: I have rewritten the script in Python, which should be a bit more solid since it parses the HTML using a DOM. The script requires libxml2dom and ipy. After configuring the parameters it can be dropped in /etc/network/if-up.d² where it should run automatically.

Note that it doesn't bypass the authentication: you still need a valid account in order to gain access. ↩
Make sure not to use a dot in the filename though. ↩