Tuesday, February 18, 2014

Running background tasks with Fabric

Using Fabric to run remote commands when administering servers, is definitely quite a time saver. One of the issues that I keep stumbling across is running background jobs on a remote machine. For example, one of the things we do to test code at work is to create an Amazon AWS instance with the application deployed on it and then run browser integration tests against that instance. If we were to just use the regular run() or sudo() command to run a long running test, we would have to make sure that we don't lose the connection between our local development machine and the AWS instance (otherwise the tests would just stop running). So, clearly, we want to start the long running task and then be able to go on our merry way.

As you may know, there are several ways to start background jobs on a *NIX like machine: nohup, screen, etc. The issue is that running things in the background may cause some issues when using Fabric. Just take a look at the Fabric FAQ that covers this topic, along with this nice discussion.

The easiest solution that I found that works in pretty much all cases is to use the one suggested here using dtach. I slightly extended the suggested solution there, to be a bit more well-rounded and complete. As you can see, this uses apt-get to install dtach, so if you are not running Ubuntu, make sure to update that appropriately.
from fabric.api import run
from fabric.api import sudo
from fabric.contrib.files import exists


def run_bg(cmd, before=None, sockname="dtach", use_sudo=False):
    """Run a command in the background using dtach

    :param cmd: The command to run
    :param output_file: The file to send all of the output to.
    :param before: The command to run before the dtach. E.g. exporting
                   environment variable
    :param sockname: The socket name to use for the temp file
    :param use_sudo: Whether or not to use sudo
    """
    if not exists("/usr/bin/dtach"):
        sudo("apt-get install dtach")
    if before:
        cmd = "{}; dtach -n `mktemp -u /tmp/{}.XXXX` {}".format(
            before, sockname, cmd)
    else:
        cmd = "dtach -n `mktemp -u /tmp/{}.XXXX` {}".format(sockname, cmd)
    if use_sudo:
        return sudo(cmd)
    else:
        return run(cmd)

Capturing the output

Although the above snippet works perfectly fine if you are just running a background task and then wanting to forget about it, what happens if you want to capture the output from some command? The problem is that you can't just redirect the output of dtach like you would with nohup. The simplest solution that I could come up was to make dtach run a bash command and explicitly redirect the output. So, I have another function that helps me accomplish this.
def run_bg_bash(
        cmd, output_file=None, before=None, sockname="dtach", use_sudo=False):
    """Run a bash command in the background using dtach

    Although bash commands can be run using the plain :func:`run_bg` function,
    this version will ensure to do the proper thing if the output of the
    command is to be redirected.

    :param cmd: The command to run
    :param output_file: The file to send all of the output to.
    :param before: The command to run before the dtach. E.g. exporting
                   environment variable
    :param sockname: The socket name to use for the temp file
    :param use_sudo: Whether or not to use sudo
    """
    if output_file:
        cmd = "/bin/bash -c '{} > {}'".format(cmd, output_file)
    else:
        cmd = "/bin/bash -c '{}'".format(cmd)
    return run_bg(cmd, before=before, sockname=sockname, use_sudo=use_sudo)
As you can see, all this does is that it wraps the command with an explicit call to bash which then is the one that interprets the output redirection. That's it! Happy dtaching!