2008

My Drupal setup

Seeing that I've spent countless hours setting up my Drupal installation, I thought that I would share this with others and document it for future reference. Drupal is an extremely powerful CMS which can be used to create a wide variety of sites. The disadvantage of this is that it requires a fair amount of work to setup a straightforward blog, which involves installing and configuring numerous modules.

Installation

Since there is no Ubuntu package for Drupal 6, I created my own package based on the drupal5 one. I set it up as a virtual host in Lighttpd by simply symlinking the domain name to /usr/share/drupal6. I created a MySQL database for the site and went through the Drupal install process. Since I'm using a multi-site installation, I also needed to alias the /files directory for each site.

$HTTP["host"] == "michael.gorven.za.net" {
    alias.url = ( "/files/" => "/home/mgorven/public_html/" )
}

Clean URLs

Clean URLs allows one to have URLs like /about instead of /index.php?q=about. This however requires that the web server rewrites URLs from the former to the latter. Drupal includes an htaccess file containing settings for Apache, but not for Lighttpd. Lighttpd does have a rewrite module, but it doesn't support the conditions that Drupal needs (such as checking if a file exists).

Lighttpd does however have a module which allows one to add scripts to the request process written in [Lua][]. A script has already been [developed][drupal.lua-devel] which implements the required rewriting for Drupal. The following lines in lighttpd.conf enable this for a specified site (after enabling [mod_magnet][] and downloading the [script][drupal.lua]).

$HTTP["host"] == "michael.gorven.za.net" {
    index-file.names = ( "index.php" )
    magnet.attract-physical-path-to = ( "/etc/lighttpd/drupal.lua" )
}

Blog

When Drupal is first installed, there is no mention of blogging as such. The first step is to enable the Blog core module1. This creates a blog content type and enables a blog for each user. (The module is designed for a multi-user blog, but can be used for a single user as well.) However, this doesn't give you all the functionality you expect from a blog engine.

Tagging is handled by the Taxonomy core module. You first need to create a vocabulary though, and enable it for blog posts. (This took me ages to discover.) In order to get nice URLs (including the date and post title, for example) you need to install the [Pathauto][] module and configure a pattern for blog posts. You may also want to define a pattern for tags.

There is also no archive functionality. The best way that I can find is the [Views][] module. It includes a pre-defined "archive" view which can display posts from a specific month, and links to the monthly pages. Even after much investigation I couldn't get the archive to behave like typical blog archives (i.e. /blog/2008/07/07, /blog/2008/07 and /blog/2008 for daily, monthly and yearly archives respectively).

Other Blog Features

The [Trackback][] and [Pingback][] modules implement automatic linking with other blogs. (I haven't actually tested these yet.) The Blog API core module allows the blog to be managed with external clients. The [Markdown][markdown-mod] module allows you to write posts using [Markdown][] syntax instead of HTML.

Comments

Drupal enables comments for blog posts by default. The [Akismet][akismet-mod] module implements spam filtering using the [Akismet][] service. The [CAPTCHA][captcha-mod] and [reCAPTCHA][recaptcha-mod] modules allows you to require users to answer a [reCAPTCHA][] when submitting comments. (I haven't actually enabled [CAPTCHAs][captcha] since I haven't gotten any comment spam yet. Or real comments for that matter...)

Posting by email

The [Mailhandler][] module allows you to submit posts via email. The configuration is fairly straightforward, except for the available commands which can be found [here][commands]. These can be specified at the beginning of emails and in the configuration as defaults. I use the following commands.

type: blog
taxonomy: [mail]
promote: 1
status: 1
comment: 2

This creates blog posts and tags them with the "mail" tag. Posts are published and promoted to the front page, and comments are enabled.

The one thing it doesn't handle is attachments (such as images). There are a couple of modules2 which support this, but they aren't available for Drupal 6 yet. ([Vhata][] has also hacked together a [photo blogging][phoblog] system, but this isn't implemented as a Drupal module.) I don't really need this feature, so I'm going to wait until these modules are updated.

OpenID

The [OpenID][openid-mod] module allows you to log into your site using [OpenID][]. The [OpenID URL][openidurl] module allows you to delegate your Drupal site as an OpenID by specifying your provider and/or your [Yadis][] document.

Yadis Advertisement

Yadis documents are advertised with a meta header in the HTML document, but this isn't the ideal method of doing so since the relaying party needs to download the entire HTML file. The [preferred methods][intertwingly] are to insert an X-XRDS-Location in the HTTP headers, or to automatically serve the Yadis document if the user agent specifies application/xrds+xml in the Accept header.

The former method can be accomplished with the setenv module for Lighttpd. The second is essentially a conditional rewrite, and so requires some Lua scripting again. The following script will do the job.

if lighty.request["Accept"] == "application/xrds+xml" then
    lighty.env["uri.path"] = "/files/yadis.xrdf"
end

The following lines in lighttpd.conf will announce the Yadis document for the root URL.

$HTTP["url"] == "/" {
    magnet.attract-raw-url-to = ( "/etc/lighttpd/yadis.lua" )
    setenv.add-response-header = ( "X-XRDS-Location" => "http://michael.gorven.za.net/files/yadis.xrdf" )
}

Random Stuff

The tag block is generated by the [Tagadelic][] module. The "Recent Tracks" block is generated from my [LastFM][] feed by the Aggregator core module, and the list of networks is simply a custom block. The [Atom][] feed is generated by the [Atom][atom-mod] module. The contact form, search and file upload are all core modules.

Missing Stuff

The one thing I haven't sorted out is image handling. There are a couple ways to [handle images][drupal-images] in Drupal, but none of these appeal to me (they're too complicated). I will probably just upload images as attachments and insert them manually in the body.


  1. It is however possible to run a blog without the Blog module. 

  2. [Mailsave][] and [Mobile Media Blog][mmb]. 

Gmail-like mail setup

I have been using Gmail for a while now, and really think that it's about the best email provider out there. I recently moved my mail over from Google Apps to my own server, but I wanted the major features that I liked. I've always used a desktop mail client and used POP3 and SMTP to receive and send mail.

These are the features I particularly like:

  1. Secure access with TLS/SSL
  2. Outgoing SMTP with authentication
  3. Messages sent via SMTP are automatically stored in the mailbox
  4. Messages downloaded via POP3 are still stored on the server
  5. IMAP and Web access

I therefore set out to recreate this setup as closely as possible. The first two are satisfied by a standard Postfix setup with TLS and SMTP AUTH. The last one is done with Dovecot and Roundcube.

To automatically store sent messages on the server, I used Postfix's sender_bcc_maps to BCC messages I send to myself, and the following Procmail recipe to move these messages to the Sent folder.

:0
* ^Return-Path.*me@example.com
.Sent/

To make POP3 access independent from IMAP, I first configured Dovecot to use a different mail location for each as follows.

protocol imap {
    mail_location = maildir:~/Maildir
}
protocol pop3 {
    mail_location = /var/mail/%u
}

I then used the following Procmail recipe to send incoming messages to both locations.

DEFAULT=$HOME/Maildir/
:0c:
/var/mail/mgorven

At the moment this is only setup for my user, but it should be possible to do it for all users by creating a global procmailrc and telling Postfix to deliver all mail using Procmail. This is working fairly well. The only part missing is that Gmail can archive or mark messages as read when they are downloaded via POP3, whereas in my setup POP3 and IMAP are completely independent.

Postfix with SPF and DKIM

As most people know, email is horribly insecure. It is trivial to forge the From address in emails since there is no authentication when sending1. This means that one cannot trust the From address, and also that people can forge messages from your address. In order to address this a number of new schemes have been developed. These include SPF, DomainKeys, DKIM and SenderID. All of these aim to verify that mail is actually from the address it appears to be from. SPF and SenderID do so by restricting which hosts are allowed to send messages from a certain domain, while DomainKeys and DKIM use cryptographic signatures.

Unfortunately all of these schemes have problems due to the fact that they are an addition to the existing mail system. SPF and SenderID prevent plain forwarding (requiring additional schemes like SRS or whitelisting of forwarders), and MTAs and mailing lists which modify messages break DomainKey and DKIM signatures. Despite these issues, email forgery is an issue which needs to be addressed, and we cannot wait for a perfect solution before adopting it. Some major mail providers (including Gmail and Yahoo) are already implementing these schemes.

I have therefore configured SPF and DKIM in my Postfix mail setup. My SPF policy allows mail from my server and SOFTFAILs all other hosts, and all outgoing mail is signed with DKIM. Incoming mail is checked for SPF and DKIM, but aren't discared even if the checks fail. I will be keeping an eye on things and will revise my policy when I think it safe.

SPF Configuration

To create an SPF policy, add a TXT record to your DNS records according to the SPF syntax. The policy should authorise all hosts from which you send mail. (Mine simply authorises my mail server since I send all mail through it.) You also need a policy for the hostname presented by your mail server in its HELO/EHLO command. You should also create policies for all subdomains which aren't used for mail.

To check SPF records for incoming mail, I used the SPF policy daemon for Postfix. It is packaged for Ubuntu as postfix-policyd-spf-python. Simply follow the instructions in /usr/share/doc/postfix-policyd-spf-python/README.Debian2, and set defaultSeedOnly = 0 in the configuration file if you don't wish to reject mail which fails the test. Remember to whitelist any servers which forward mail to you (i.e. you have another address which gets forwarded to your mail server), unless they implement SRS3.

DKIM Configuration

To sign and check DKIM I use DKIMproxy. There isn't an Ubuntu package so I installed it from source. The instructions on the site are good, and include details for Postfix. You will need to generate a key to sign with and publish it in DNS, and then configure Postfix to sign outgoing messages and validate incoming messages. DKIMproxy won't discard messages with invalid signatures by default.

DKIM includes a component called ADSP which allows domains to publish their signing policy. The strongest policy states that all messages are signed with DKIM and any messages without signatures should be discarded. This will allow mail servers to reject messages not sent through your mail server. However, the standard is not finalised yet, and issues regarding mailing lists still need to be addressed.


  1. Yes, I know about SMTP authentication, but people can simply use a relay. 

  2. Just watch out for the location of the configuration file -- the README uses a different location to the package. 

  3. Gmail doesn't implement SRS as such, but does use a compatible rewriting scheme. 

First Post

So, I have finally jumped on the blogging bandwagon. I'm not quite sure that it's the wagon for me, but I'll give it a try anyway. Most of the content will be fairly technical, since I'm not a pour-your-heart-out type of guy. A lot of it will be about things I setup on Linux, occasionally some code which I've written, and possibly some stuff about my Masters project.

This site is powered by Drupal, Lighttpd, MySQL and Ubuntu. This is all running on a Xen virtual machine. Many thanks to Vhata for the awesome deal.