A blog mostly about Python.

2010-03-16

Buildbot and nose test coverage

I have been using buildbot (and nose) for a little while now and I really like it (I like nose too, just for the record). It's very hackable and I think I will be able to accomplish most, if not all, of the things I want to do with it. One of those things is to get the test coverage displayed and easily accessible from buildbot's web status application. Like so:
Can you see the little link with the text "coverage 84%"? Clicking that link will take you to the nice HTML report generated by nose (via coverage.py) during the test run.

To use this little hack you will need a buildbot slave with access to a directory that gets published by some web server. The custom build step below puts the coverage report directly into that directory and then adds an URL to the report, that will be visible in buildbot, with a call to addURL().
import re
from buildbot.process.properties import WithProperties
from buildbot.steps.shell import ShellCommand

class Nose(ShellCommand):
    def __init__(self):
        self.coverage_path = '/var/www/foo/coverage-%s'
        self.coverage_url = 'http://example.com/foo/coverage-%s'
        d = WithProperties('--cover-html-dir=' + self.coverage_path, 
                           'buildnumber')
        command = ['nosetests', '--with-coverage', '--cover-erase', 
                   '--cover-html', d]
        ShellCommand.__init__(self, command=command)
    
    def describe(self, done=False):
        if not done:
            return 'testing'
        else:
            return 'test done'

    def createSummary(self, log):
        buildnumber = self.getProperty('buildnumber')
        coverage_index = (self.coverage_path + '/index.html') % buildnumber
        f = open(coverage_index)
        m = re.search(r'Percent: (\d+) %', f.read())
        f.close()
        self.addURL("coverage %s%%" % m.group(1), 
                    self.coverage_url % buildnumber)
Good enough for now.

An even nicer version of this build step would download all the coverage report files from the slave to the master. The master would then serve the files using it's web status application. This would eliminate the need for a web server on the slave. But that's the deluxe version, more on that later.

5 comments:

  1. Just wanted to say thanks, you've saved me a lot of work.

    ReplyDelete
  2. Very lovely post that helped me!
    I've blogged about doing the same when the coverage files aren't available to the master (since createSummary runs on the master and not the slave) in case that helps anyone: http://bit.ly/9tOayv

    ReplyDelete
  3. Hi,
    I am a beginner and using buildbot to automate my build/execute process for libc.
    I wanted to use addURL() method for custom URLs at the build status page. I read the manual but couldnt understand it.!
    Please give me a simple example how to accomplish it.?
    Ex. If I do,
    {
    class Test(BuildStep):
    def createlink(self, log):
    url = "http://www.google.com"
    self.addURL("coverage", url)

    f1 = factory.BuildFactory()
    f1.addStep(shell.ShellCommand( blah blah ))
    }

    I want to use custom url for the above f1.addStep()
    Please guide !!

    Thanks,
    Nikunj

    ReplyDelete
  4. Mattias, Hi - finally found a way to reach you.
    I found your answer here: http://stackoverflow.com/questions/531941/how-to-setup-google-c-testing-framework-gtest-on-visual-studio-2005/4222169#
    and wanted to ask you a question but could not do it on stackoverflow.
    In that answer you mention that: "Add gtest.lib and gtest_main.lib to Common Properties->Linker->Input->Additional Dependencies"
    I have done similar thing and it is working, just my question is do you add these for the Debug Mode or Release Mode? Or Both?
    With me it works for the Debug Mode.
    Thank you. Looking forward for your reply.

    ReplyDelete
  5. Hi,but the publishing and transaction environments that sit behind the site are sympathetic to the needs of the business management processes in Web Design Cochin as well as the needs of those who must support and maintain the site.Thanks..........

    ReplyDelete

Followers