My fiancée and I recently bought our first property, which was an interesting experience. It is a fairly complicated process, and many people don't understand how it works or the costs associated with it. So I thought that I would document it to help anyone else looking to purchase property for the first time.

Deciding what you need

The first thing I suggest you do is decide what you need and what you want. Make a list of hard requirements, and another list of nice-to-haves. Consider things like: number of rooms, secure parking and garages, automatic gates, baths and showers, flat or townhouse or freestanding, garden, pool, balcony and appliance space.

Determining what you can afford

Work out how much of your income you could put into a bond each month, and use an online calculator to calculate the loan amount which that represents. Note that the maximum monthly repayment is usually limited to 30% of your income. If you are buying with your partner, your combined income can be used. Interest rates are usually linked to the prime lending rate set by the Reserve Bank. When we applied for our bond we were offered +1.2%, +0.2%, +0.05% and -0.62%, so I'd suggest working with an interest rate of prime or slightly above.

Work out what deposit you can afford, which is payable when your offer is accepted. Bear in mind that there are significant other costs which you will also need to cover (up to 5% of the purchase price). Your deposit will affect the strength of your offer, as well as your bond applications. Aim for at least 5%, and ideally 10%.

Estate agents

Most properties are sold through estate agents. The seller will approach one or more estate agents and grant them mandates to try to sell the property. The estate agent will usually perform a valuation and help the seller to choose an asking price. If (and only if) the agent manages to sell the property, the seller pays them a commission. The going rate seems to be 7.5% of the selling price (excluding VAT).

Visiting properties

The best way to discover properties still seems to be the property section of the newspaper, although some of the estate agents have semi-decent website listings. We spent four Sundays visiting showhouses listed in the newspaper, as well as a few others which we made appointments to see. We probably visited about 25 properties in total, which is fairly good going. I would suggest visiting at least 10 properties before putting an offer in — it takes a while to be able to judge the value of a property.

Inspection

When you find a property you really like and are seriously considering putting in an offer, I suggest that you go back for a more detailed inspection. I'm not suggesting that all these conditions be met, but that they should be considered when determining how much the property is worth to you.

• How much cupboard space do the bedrooms have?
• What state is the bathroom in?
• How old is the geyser?
• What's the water pressure like?
• Do the taps work properly?
• How big is the kitchen?
• How many appliances is there space for (washing machine, tumble dryer, dishwasher)?
• What state are the oven and hob in?
• What state are the kitchen and bedroom cupboards in?
• How new is the distribution board, plugs and light fittings?
• Is the electricity prepaid?
• Is there an alarm?
• Do all the windows have burglar guards?
• What condition are the doors and windows in?
• Are there security gates on the doors?
• What condition is the wall and ceiling paint in?
• What condition is the roof in?

Ask why the owner is selling, and what crime in the area is like. Check what the monthly rates are (preferably get a copy of the latest rates statement).

Sectional title

Check how many units there are (the fewer the better), and how many have live-in owners versus tenants (more live-in owners is better). Ask what the body corporate is like and how effectively it operates. Get the latest financial statements of the body corporate and check that they are financially sound. Get and read the conduct rules of the scheme (are pets allowed?). Get the latest levies statement and check what the levies are¹. Ideally get the sectional title plan (a diagram showing the units) and check that the unit number is correct.

Making an offer

Once you've found a property you want to buy, you make an offer to the seller. This is a formal process where you draw up an Offer to Purchase which sets out the details and conditions of the offer. The seller will have a certain amount of time in which to accept the offer (usually about 2 days), and once they have signed the document it becomes the Agreement of Sale. The seller could also come back with a counter offer.

Remember that the asking price is simply a guide, and also that the agent wants to sell the property for as much as possible (since their commission is linked to the selling price). You need to decide how much the property is worth to you. In most cases it's worth starting with a lower offer and raising it if it's rejected.

The offer defines when occupation is handed over, either on a specific date, or when transfer goes through (i.e. when you legally own the property). If occupation is handed over before or after transfer, then the party with occupation will owe the owner an occupational levy (rent, basically).

Applying for a bond

Once your offer is accepted you will need to apply for a bond. It's worth applying to multiple lending institutions in order to get the best deal. The recent trend is to use a mortgage originator, who applies to numerous institutions on your behalf. They earn commission from the institution if a bond is accepted, so theoretically² there's no cost to you. What we did was apply directly to the bank which we bank with, and got the mortgage originator to apply to the other institutions. This process usually takes about 2 weeks.

The lending institution charges an initiation fee on the bond, and there is usually a monthly service charge as well. The institution will require that the building is insured, although with sectional titles this is done by the body corporate.

Conveyancing

Conveyancing is the process of transferring ownership of the property, and is performed by a legal firm. The buyer is usually liable for the attorneys' fees³, as well as various costs they incur (rates clearance certificate from the municipality, levy clearance certificate from the body corporate, and deeds office fee). The conveyancers also collect transfer duty from the buyer, which is a tax levied by the government. The legal firms agree on a recommended fee structure for their services, but are not obligated to stick to it. This process usually takes about 8 weeks.

Traditionally the conveyancers are chosen by the seller, which is a pretty stupid system since the buyer pays them. You may get a better deal if you shop around for conveyancers before hand and nominate them in your Offer to Purchase.

Bond registration

The mortgage bond needs to be registered against the property, and this is also done by a legal firm. Similar to the conveyancers, the buyer is liable for the attorneys' fees and associated costs. The bond attorneys are usually nominated by the lending institution. Like the conveyancing, there is a standard fee structure, but firms are not obligated to actually use it.

Transfer

The conveyancers and bond attorneys lodge the various papers at the deeds office simultaneously, and it then takes a week or two to process the transfer. All in all the whole process should take about 3 months to complete. You will be liable for all rates and levies from the date of transfer.

Tools

Ooba have an affordability calculator to calculate what bond you can afford, and a bond and transfer costs calculator to estimate the purchase costs. Fin24 have a good bond calculator which graphs the capital and interest components of the bond.

I have also written my own purchase costs calculator and bond calculator, which should give similar answers.

After over a year of development, we have finally released Ibid. Ibid is a general purpose chat bot written in Python. We've suffered from a bit of feature creep, so despite being a 0.1 release it can talk 7 messaging protocols and has over 100 features provided by plugins. I think we also have an excellent architecture and very developer friendly plugin API. The 0.1.0 release can be downloaded from Launchpad or installed from our PPA.

I am replacing my current PGP key, 6612FE85, with a new key, 1E016BE8, as of 1 July 2009. A signed version of this announcement can be found here.

5FE6 76B9 9696 DB6E 0B2B  B2A7 2956 B173 1E01 6BE8

SuperGenPass and Password Composer are password generators, which generate different passwords for each site you use based on a single master password. This gives you the convenience of only remembering one password as well as the security of using different (and strong) passwords for each site. This means that you won't have all your accounts compromised when¹ one of them is compromised.

Most password generators are implemented as browser extensions or bookmarklets, since they are most frequently needed in a web browser. I've been wanting to start using a password generator, but I wanted to be sure that I could access my accounts even if I didn't have a web browser accessible. The two situations I could think of were a command line only system (e.g. SSH) and my cellphone².

Surprisingly, I couldn't find a command line implementation of SuperGenPass, so I wrote one in Python. I also couldn't find any J2ME or Symbian implementations, and so wrote my own one in J2ME. They both support subdomain stripping and configurable password lengths. They don't support salted passwords.

I chose SuperGenPass over Password Composer because it uses a better scheme. Password Composer only uses hex characters, whereas SuperGenPass uses a base64 encoded hash. SuperGenPass also hashes the password multiple times (which would slow down a brute force attack to find the master password) and imposes complexity requirements on the generated password (which reduces the chances that the generated password can be brute forced).

Following on from yesterday's post, I decided to try implement proper content negotiation. After a fair amount of time spent getting to grips with Lua, I got a script which works very nicely. It implements server driven content negotiation for media types.

The basic idea of content negotiation is that a resource (e.g., this graph) exists in multiple formats (in this case, SVG, PNG and GIF). When a user agent requests the resource, it indicates which formats it understands by listing them in the Accept header. The server compares these to the available formats and sends the best one. So a browser which can display SVG will receive the diagram in SVG format, while a browser which can't will receive it in PNG (or GIF) format.

(The following description assumes knowledge of the Accept header format.)

The script works by searching the directory for files with the requested name but with an additional extension (each of which is a variant). The media type is inferred from the extension using /etc/mime.types, and the quality of the type is set by a hardcoded table in the script. Each variant is checked against the acceptable types sent by the user agent, and the overall quality calculated by multiplying the quality with the q parameter in the Accept header. The variant with the highest overall quality is then chosen.

Some browsers include wildcard entries such as image/* and */* in the Accept header without specifying a q parameter. This parameter defaults to 1 (the highest value), which means that no preference is actually indicated. The script implements the same hack that Apache does in order to compensate for this. It also handles directory index files by defaulting to files named "index".

To install the script, download and save it somewhere (such as /etc/lighttpd/). Then add the following to the site definition.

magnet.attract-physical-path-to = ("/etc/lighttpd/negotiate.lua")

URLs shouldn't really contain file extensions (like .html, .png) since they are supposed to identify a resource and not a particular representation/format thereof. The format is indicated by the Content-Type header sent in the response. Modern CMSs do this already (for example, the URL of this page doesn't include .html).

Doing the same for static files (i.e. files served directly by the webserver) isn't straightforward because most webservers use the file extension to determine the MIME type to send in the Content-Type header. This means that simply removing the file extension from the filename (or even creating a symlink without a file extension) will cause the webserver to send the wrong Content-Type header.

I decided to try find a solution to this for my webserver of choice, Lighttpd. Lighttpd has a module which embeds a Lua interpreter and allows you to write scripts which modify (or even handle) requests. So I wrote a script which searches the directory for files with the same name as requested but with an extension. This means that any file can be accessed with the file extension removed from the URL while still having the correct Content-Type.

The script currently chooses the first matching file, which means that having multiple files with the same name but different extensions doesn't do anything useful. The proper method however is to actually do content negotiation, which chooses the format based on the preferences indicated by the HTTP client in the Accept header.

To use this script, download it and save it somewhere (I use /etc/lighttpd/). Enable mod_magnet, and add the following line to the site definition.

magnet.attract-physical-path-to = ("/etc/lighttpd/extension.lua")

I recently came across an article which tries to address the problem of SSH agent forwarding with screen. Briefly, the problem is that reattaching a screen instance breaks agent forwarding because the required environment variables aren't present in the screen instance. The solution given didn't quite work for me though because I use an SSH wrapper script which automatically runs screen.

My solution is to write a screen wrapper script which stores the environment variables in ~/.sshvars (as export statements) and then starts screen. Running source ~/.sshvars in the shell then makes the variables available. (I created an alias called fixssh to do this.)

I like to put wrapper scripts in ~/bin with the same name. This didn't work out the box however, since ~/bin is only added to PATH in ~/.profile but this file is only sourced if the shell is interactive. The fix is therefore to add the following to ~/.bashrc, but near the top before the [ -z "$PS1" ] && return line.

if [ -d "$HOME/bin" ]; then
    PATH="$HOME/bin:$PATH"
fi

One of the most effective anti-spam measures one can implement is to enforce valid use of SMTP and other standards. Spam clients are interested in sending messages as quickly as possible, and so usually don't bother with actually implementing standards correctly. In this post I shall describe the various checks which can be used, show how to implement these checks in Postfix, and describe how to ensure that your mail server passes these checks when sending mail.

Reverse DNS Entries

RFC 1912 states that "every Internet-reachable host should have a name" and "make sure your PTR and A records match". This can be checked by performing a Forward Confirmed reverse DNS lookup¹. This check can be done before even accepting the TCP connection, which means the mail server's existence isn't even revealed to rejected clients.

Postfix: Add reject_unknown_client_hostname to smtpd_client_restrictions.
Passing: Ensure that your mail server has matching PTR and A records.

HELO/EHLO Hostname

RFC 2821 states that "a client SHOULD start an SMTP session by issuing the EHLO command". Almost all SMTP client implementations do do this, and so we can require the use of HELO/EHLO.

Postfix: Set smtpd_helo_required = yes.

RFC 2821 states that "the argument field [of the HELO/EHLO command] contains the fully-qualified domain name of the SMTP client if one is available". Since external mail servers have to be Internet reachable this is a requirement, and can be checked by looking up the name in DNS².

Postfix: Add reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname and reject_unknown_helo_hostname to smtpd_helo_restrictions.
Passing: Ensure that your mail server is configured to send a fully qualified hostname which exists in DNS.

If there is only one mail server (and possibly even if there are multiple servers), SMTP clients should not be using the server's hostname as the HELO hostname. Clients which do so can therefore be rejected.

Postfix: Add check_helo_access hash:/etc/postfix/helo_access to smtpd_helo_restrictions. Use helo_access as a template for /etc/postfix/helo_access, and run postmap /etc/postfix/helo_access afterwards.

Originator Address

The originator (MAIL FROM) address is where error reports will be sent, and therefore should be a valid address. The only thing which can be checked though is that the address is fully qualified and that the domain exists.

Postfix: Add reject_non_fqdn_sender and reject_unknown_sender_domain to smtpd_sender_restrictions.
Passing: Ensure that your mail server only emits fully qualified addresses. This should happen by default, except possibly for mail submitted with sendmail.

Recipient Addresses

Unless the mail server is a relay or backup MX, it should already only be accepting addresses for which it is the destination. If it is a relay or backup MX the same checks as above can be done.

Postfix: Add reject_non_fqdn_recipient and reject_unknown_recipient_domain to smtpd_recipient_restrictions.

One other check has to do with multiple recipients for bounced mail. Error reports for bounced mail uses a null originator address, and should only have one recipient.

Postfix: Add reject_multi_recipient_bounce to smtpd_data_restrictions.

Pipelining

Unless the client explicitly requests pipelining (as described in RFC 1854), the SMTP conversation must occur in lock step (i.e. the client must wait for a response from the server before sending the next command). Since spam clients are trying to send messages as quickly as possible it is likely that they do not adhere to this requirement.

Postfix: Add reject_unauth_pipelining to smtpd_data_restrictions.

RFC 2821 specifies that the server must send the first message after the connection is established. A neat trick is to delay this initial message to catch out clients which don't wait for it.

Postfix: Add sleep 1, reject_unauth_pipelining to smtpd_client_restrictions. This also requires smtpd_delay_reject = no (explained below).

Monitoring

Since these measures will reject valid mail from misconfigured mail servers, I like to keep an eye on rejections via logcheck. However, some of these measures by their very nature reject the client before it's even sent the originator and recipient addresses, which makes identification of valid mail difficult. Postfix therefore has a feature which delays rejection until after the recipient addresses have been sent. This is enabled by default, but can be disabled by setting smtpd_delay_reject = no.

I've recently been doing a lot of hacking on Knab, which is the software behind the #clug IRC bot, Spinach. I've contributed a number of new modules, most of which are running on Spinach and are available from the main Bazaar repository.

Events

This module is basically a calendar feature which can store and retrieve events such as birthdays. It also handles recurring events (both with rules¹ or multiple dates).

<cocooncrash> Something happens on 21 December 2008 at 15:00
<Knab> yessir
<cocooncrash> When is something?
<Knab> something is on Sunday the 21st of December 2008 at 15:00:00
<cocooncrash> Forget event something
<Knab> I've forgotten something

<cocooncrash> My birthday happens on 22 March every year
<Knab> sure
<cocooncrash> How long until cocooncrash's birthday?
<Knab> your birthday is 13 weeks and 1 day away
<cocooncrash> How many days until my birthday?
<Knab> your birthday is 92 days away

<cocooncrash> When is Easter in 2010?
<Knab> Sun 4 Apr
<cocooncrash> What's happening on Tuesday
<Knab> National Chocolate Covered Anything Day and Day of Reconciliation
<cocooncrash> Search events for Outsider 
<Knab> Found: Be Friendly To Outsider Day, Be Hard On Outsider Day, Outsider's birthday, Outsider's dad's birthday,  (showing 4 of 4 results)

Twitter

This module retrieves messages posted on Twitter and identi.ca.

<cocooncrash> Last twitter update by iolproofreader
<Knab> "The ANC in the Eastern Cape is making it clear: it's either you're with us or not."
<cocooncrash> twitter 1067135354
<Knab> iolproofreader: ""Friend chases after golfer shot" http://tinyurl.com/4dn2lx"
<cocooncrash> http://twitter.com/iolproofreader/status/1067135354
<Knab> iolproofreader: "Police are planning on more patrols and more scrutiny for travellers crossing the border of Swaziland and Mozambique."

URLLengthen

This modules discovers the URL which a shortened URL redirects to.

<cocooncrash> http://is.gd/1D
<Knab> http://google.com/

URLShorten

This module creates a short URL using is.gd.

<cocooncrash> shorten http://google.com/
<Knab> http://is.gd/1D

HTTP

This module retrieves an HTTP URL and returns the result of the request.

<cocooncrash> get http://google.com
<Knab> 200 OK "Google"
<cocooncrash> Is google.com up?
<Knab> Yes, it's up
<cocooncrash> Is Twitter down?
<Knab> No, it's just you

GoogleDefine

This module gets definitions using Google.

<cocooncrash> gdefine spinach
<Knab> "dark green leaves; eaten cooked or raw in salads  ", "The Spinach (Eulithis mellinata) is a moth of the family Geometridae.

Subversion

This module retrieves commit messages from a Subversion repository.

<cocooncrash> last commit
<Knab> r721 by trb143 7 hours ago: "Update UI a bit to display columns"
<cocooncrash> commit 700 full
<Knab> r700 by mgorven on 2008/12/15 at 21:14:26 UTC: "Set non-existing config entries if requested with a default value, and create ~/.openlp when saving config file. " M /openlp-2/trunk/openlp/core/utils/linregistry.py, M /openlp-2/trunk/openlp/core/utils/confighelper.py

SummonJabber

This module summons people by sending them a message via Jabber.

<cocooncrash> cocooncrash.summon.jabber is cocooncrash@example.com
<Knab> gotcha
<cocooncrash> summon cocooncrash
<Knab> I've summoned cocooncrash via Jabber

Feeds

This module retrieves RSS and Atom feeds.

<cocooncrash> Add feed http://www.news24.com/news24RSSFeed/0,,2,00.xml as News24
<Knab> okay
<cocooncrash> latest articles from news24
<Knab> 0: "SA's first San party launches", 1: "Pakistan mall collapse: 3 dead", 2: "Iran warship to fight pirates", 3: "Westwood storms into lead", 4: "US Embassy given the boot", 5: "Jennifer Aniston's chilli Xmas"
<cocooncrash> article 0 from News24
<Knab> "SA's first San party launches" http://www.news24.com/News24/South_Africa/Politics/0,,2-7-12_2444493,00.... : The first San political party has been launched at Upington in the Northern Cape. 
<cocooncrash> article /Palin/ from news24
<Knab> "Drama in Palin household" http://www.news24.com/News24/World/News/0,,2-10-1462_2444295,00.html : The mother of  an 18-year-old man who plans to marry Alaska Governor Sarah Palin's  pregnant daughter, Bristol, has been arrested on drug charges.
<cocooncrash> list feeds
<Knab> m&g, clug park, haiybo, TechCrunch, onion, News24
<cocooncrash> remove news24 feed
<Knab> okay

I wrote three IRC bots in Python this last week (although one was a rewrite). They probably aren't very useful to most people, but I'm going to share them anyway in case someone finds them interesting.

The first one was prompted by Adrian, who is maintaining a countdown until his wedding as a factoid in Spinach. Since Knab doesn't actually support countdowns, it has to be updated manually. This clearly isn't the Right Way to do this, and so I hacked together a script which connects to IRC and teaches Spinach the updated factoid. I run this as a daily cronjob to keep the countdown up to date.

As is usually the case with Python, there was already a library for accessing IRC, namely irclib. It isn't documented very well, but has a couple example scripts which are fairly easy to follow. It follows an event based model, so you write functions which will be called when certain events occur (such as receiving a message).

The final of the Currie Cup was held on Saturday (which my team (the Sharks) won), and I followed the match online using SuperSport's live score site¹. I then thought that it would be cool to have the score announced on IRC when it changed, and since I was bored I wrote a simple bot to do this. It worked well, but was very simple in that it only supported one hardcoded channel and one hardcoded game.

Since I was also bored on Sunday I rewrote this bot properly. I added a subscription mechanism so that channels and users can subscribe and unsubscribe to games by sending the bot a command. It's mostly working except for listing the available games (since there aren't any rugby games coming up which means that I can't test it ;-) ). Games are specified by the ID used by SuperSport's site, and finding the right ID is currently a manual process.