Python Decorators

For my Masters project I need a method by which the user can specify which functions should be run on an SPE1. This method should be simple, clear and easy to turn on and off. I stumbled upon a blog post a little while ago (I think it was this one) which explained decorators in Python, which is the perfect tool for the job. Decorators are used to transform functions, but without changing the function itself or the calls to it.

def spe(func, *args):
    def run(*args):
        return compile(func, *args)
    return run

def sub(a, b):
    return a - b

print sub(2, 4)

The spe function is the actual decorator. The @spe line applies the decorator to the sub function. Implicitly, the following declaration is made:

sub = spe(sub)

The sub function is being wrapped by the spe function, and so all calls to sub (such as the print line) will use the wrapped function instead. The decorator creates and returns a new function called run which will (eventually) cause the original function to be compiled and executed on an SPE. This means that running a function on an SPE will be as simple as adding @spe before the function declaration2, without having to change the way in which the function is called. Turning it off is as simple as commenting out this line, and it is fairly clear as to what is happening.

  1. Trying to make this decision automatically would be a massive project in itself and would probably be worse than a human decision. 

  2. There will probably be some restrictions on what the function may contain, but that's a different matter. 

Masters project overview

Since I might be posting entries regarding my Masters project, I thought that I would provide a brief overview of the project to put it in perspective. I am doing my MSc in Electrical Engineering at UCT as part of the ACE group headed by Prof. Inggs. The group is based at the CHPC, which is part of the Meraka Institute, which in turn is part of the CSIR. The group's research is focused on developing new platforms and methods for HPC.

My project is to investigate the suitability of the Cell processor for HPC. The Cell processor is found in the PlayStation 3 and in BladeCenters, and is a very powerful processor. It achieves this by using two different types of cores. The one type (PPU) is a general purpose core capable of running an operating system, while the other type (SPU) is designed specifically to crunch numbers.

The disadvantage of this architecture is that it is very difficult to program for. When using the IBM Cell SDK, the user needs to write separate programs for each type of core, and needs to manage the SPEs manually as well as take care of all memory transfers. This requires a good knowledge of the architecture, and results in a lengthy development process and unportable code.

For the Cell processor to be a successfull platform in HPC the development process must be made easier while still making efficient use of the Cell's capabilities. There are a number of commercial and non-commercial tools which aim to do this using a variety of methods. I have looked into these tools and have not found one which is both effective and open.

I therefore aim to create my own platform with which to program the Cell processor. The idea is to use [Python][] as the end user language, and to make a backend which transparently runs certain functions on the SPEs. This will involve converting the functions into C or C++, adding code to manage execution on an SPE and do the required memory transfers, compile it with the [GCC][] compiler and then execute it.

It is quite an ambitious plan, and there are a lot of potential pitfalls. If it succeeds however, I think that it will be a very easy way to develop for the Cell processor while still having portable code.

Syndicate content