Buildbot and nose test coverage

I have been using buildbot (and nose) for a little while now and I really like it (I like nose too, just for the record). It's very hackable and I think I will be able to accomplish most, if not all, of the things I want to do with it. One of those things is to get the test coverage displayed and easily accessible from buildbot's web status application. Like so:

Can you see the little link with the text "coverage 84%"? Clicking that link will take you to the nice HTML report generated by nose (via coverage.py) during the test run.

To use this little hack you will need a buildbot slave with access to a directory that gets published by some web server. The custom build step below puts the coverage report directly into that directory and then adds an URL to the report, that will be visible in buildbot, with a call to addURL().

import re
from buildbot.process.properties import WithProperties
from buildbot.steps.shell import ShellCommand

class Nose(ShellCommand):
    def __init__(self):
        self.coverage_path = '/var/www/foo/coverage-%s'
        self.coverage_url = 'http://example.com/foo/coverage-%s'
        d = WithProperties('--cover-html-dir=' + self.coverage_path, 
                           'buildnumber')
        command = ['nosetests', '--with-coverage', '--cover-erase', 
                   '--cover-html', d]
        ShellCommand.__init__(self, command=command)
    
    def describe(self, done=False):
        if not done:
            return 'testing'
        else:
            return 'test done'

    def createSummary(self, log):
        buildnumber = self.getProperty('buildnumber')
        coverage_index = (self.coverage_path + '/index.html') % buildnumber
        f = open(coverage_index)
        m = re.search(r'Percent: (\d+) %', f.read())
        f.close()
        self.addURL("coverage %s%%" % m.group(1), 
                    self.coverage_url % buildnumber)

Good enough for now.

An even nicer version of this build step would download all the coverage report files from the slave to the master. The master would then serve the files using it's web status application. This would eliminate the need for a web server on the slave. But that's the deluxe version, more on that later.

Git post-receive (commit emails) hook in Python

I recently needed to host a private Git repository on my home server. I also wanted to send commit emails to the two other project members. We all have our email hosted on various servers around the net and a convenient way to send emails to us is to use Gmail's SMTP servers. To be able send emails using those server you need to authenticate with them over TLS.

The Git package on Ubuntu (and probably other distributions as well) ships with a script called post-receive-email. This is a shell script that uses sendmail to deliver emails. Now, I am no friend of sendmail. I find it much to complex to configure and get working for my taste. After a bit of searching on the net I failed to find any suitable replacement script. A suitable replacement script would be something that didn't use sendmail and is preferably written in Python. I did find a script written in Ruby but that didn't please me either. Mainly because I don't use Ruby and did not want the headache of trying to get that script to work again when I, at some point in the future, have to reinstall my server.

But it so simple to send emails using TLS in Python:

import smtplib

server = smtplib.SMTP('smtp.gmail.com', 587)
server.ehlo()
server.starttls()
server.ehlo()
server.login('fromuser@gmail.com', 'mypassword')
server.sendmail('fromuser@gmail.com', 'touser@example.com', 'HELLO')
server.rset()
server.quit()

Great, This looks like an excuse to write my post-receive script in Python. So I did! If you are in a similar need to send Git commit emails, authenticating using TLS and with a Python script (or just curious), then please feel free to take a look at my script: http://github.com/brasse/post_receive_email.py. 

Nested with statements in Python 3.1 are nice

Before Python 3.1 a convenient (but problematic) way of using nested with statements was to use the standard library provided context manager contextlib.nested. This context manager suggests that it can be used when writing code such as:

with contextlib.nested(A(), B()) as a, b:
    a.foo()
    b.foo()

As always, the devil is in the details. Since the the calls A() and B() occur before the nested context manager has been created the result of the code above is more like this:

a = A()
b = B()
with contextlib.nested(a, b):
    a.foo()
    b.foo()

This means that if an exception is raised when B() is executed the __exit__ method of a will never get called and we are now leaking resources.

Fortunately this has been fixed in Python 3.1 (and will also be in 2.7 I believe). From now on support for nested with statements is in the Python language it self. Instead of using contextlib.nested we can now do this:

with A() as a, B() as b:
    a.foo()
    b.foo()

And this really is equivalent to two nested with statements:

with A() as a:
    with b() as b:
        a.foo()
        b.foo()

Nice, I say!