Sunday, October 28, 2012

Python Function Decorators

One of the (many) neat things about Python is the fact that functions are first class objects (like in JavaScript). As the first class nature of functions has been covered many times elsewhere (take a look at this and this), I shall not talk about what that means. However, one of the things one can do with first class functions is pass them as an argument to another function. The ability to do this is what enables the use of function decorators. Again, if you really want to understand how decorators work, why the work, etc., then I suggest you read this excellent explanation. If however, you just want to see an example so that you can use it in your own code, keep reading.

There are two types of decorators, those without arguments (e.g @mydecorator) and those with arguments (e.g. @mydecorator('foo', 'bar'). For the case with no arguments, the code for your decorator will look as follows:
import functools

def mydecorator(f):
    """A simple decorator."""
    @functools.wraps(f)
    def wrapper(*args, **kwargs):
        print 'Before %s in wrapper' % f.__name__
        result = f(*args, **kwargs)
        print 'After %s in wrapper' % f.__name__
        return result
    return wrapper

@mydecorator
def myfunc():
    print 'Inside myfunc'

In case you are curious, we don't technically need to use the @functools.wraps decorator in our decorator, but it performs some very useful things for us. Take a look at this post if you want to know why we 'need' it.

If you want to pass arguments to your decorator, then your code will need be something like this:
import functools

def mydecorator(arg1, arg2):
    """A simple decorator with arguments."""
    def wrap(f):
        @functools.wraps(f)
        def wrapper(*args, **kwargs):
            print 'Decorator arguments: %s, %s' % (arg1, arg2)
            print 'Before %s in wrapper' % f.__name__
            result = f(*args, **kwargs)
            print 'After %s in wrapper' % f.__name__
            return result
        return wrapper
    return wrap

@mydecorator('foo', 'bar')
def myfunc():
    print 'Inside myfunc'


It is very important to note that the arguments passed to the decorator are only evaluated once, the first time that the function is encountered. This essentially means that you cannot change the parameters to your decorator at runtime and if arg1 was 'foo', then it will be 'foo' until your program terminates. If you can look at the code above and understand why that is at first glance, then that is admirable. If you are like the rest of us, check out the great explanation of decorators here.

Thursday, October 25, 2012

Python Project Directory Structure

The "proper" way to setup a Python project is (unfortunately) subject to some debate (just check out these Stackoverflow threads here, here, and here or search for "python project directory structure"). One of the biggest points of contention is where to put the unit tests -- should they go under the project source tree or should the be separate?

Although arguments can be made for either organization structure, I feel that following the lead of the Django project, and separating the tests from the rest of the source code is much cleaner. Doing so clearly differentiates between the actual code of the project and the testing of that code (which can be thought of as a consumer of the project code). Also, make sure not to call the directory "test" as there is an internal Python module named test.

MyProject/
    setup.py
    myproject/
        __init__.py
        somemodule.py
        somepackage/
            __init__.py
            foo.py
    docs/
    tests/
        runtests.py
        unittests/
            __init__.py
            ...
        integrationtests
            __init__.py
            ...

I'm sure that there are many who will say that I'm wrong, but at least I can fall back on the fact that Django does it this way. If you don't believe me, just check out their source code.

Monday, October 22, 2012

Splitting Python Packages Into Multiple Projects

I recently had to split up a python package into multiple separate projects while still keeping the topmost namespace the same. I.e. I had to create "namespace packages". So, a project that was initially setup like this

mypackage/
    __init__.py
    module1/
        __init__.py
        foo.py
        bar.py
    module2.py
    module3/
        __init__.py
        bar.py
        baz.py

had to be split up into something like
project1/
    mypackage/
        __init__.py
        module1/
            __init__.py
            foo.py
            bar.py
        module2.py

project2/
    mypackage/
        __init__.py
        module3/
            __init__.py
            bar.py
            baz.py
Although I was able to do this rather simply following the suggestions found here, I put together some example files so that you can quickly see how it works. Basically, the one thing that needs to be done is that in project2/mypackage/__init__.py needs to contain
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
Once this is done you can now distribute each of the projects separately while still making use of all modules and classes. For example, if we are in the directory that contains project1 and project2, we can run something like the following:
# Add the two projects to the path
import sys
sys.path.append('project1')
sys.path.append('project2')

# Import the modules from the two packages
import mypackage.module1
import mypackage.module2
import mypackage.module3

if __name__ == '__main__':
        # Create some objects from the first package
        p1_foo = myproject.module1.foo.Foo()
        p1_bar = myproject.module1.bar.Bar()
        p1_SomeClass = myproject.module2.SomeClass()

        # Create some objects from the second package
        p2_bar = myproject.module3.bar.Bar()
        p2_baz = myproject.module3.baz.Baz()

        # Show that they all work
        p1_foo.run_me()
        p1_bar.run_me()
        p1_SomeClass.run_me()

        p2_bar.run_me()
        p2_baz.run_me()
You can download all of the above code from github.