Understanding Python Decorators

Reading time: 13 minutes Published date: 22 Jun 2019

Table of Contents

Python decorators seem to be one of those mysterious parts of python code that other people write, and you just use. Here, we’ll be building a simple, but useful decorator from scratch and adding new functionality step by step to try and gain a better understanding of how decorators work, and how they can be used.

For reference, python wiki provides an excellent library of python decorator patterns, from which we will steal the Retry example. This decorator retries a method if it returns False, until the max number of retries are attempted.

0. The Basics

Before we can dig into decorators, I assume we all know the following to be True:

This means the language supports passing functions as arguments to other functions, returning them as the values from other functions, and assigning them to variables or storing them in data structures [source]

To illustrate, lets define a soccer Player class implementing some methods a player is likely to perform.

class Player():
    def __init__(self, name):
        print('%s steps onto the pitch' % name)
    def kick(self):
        print("The ball went flying over the top of the goal")
    def fall(self):
        print("A loud crash was heard")
    def score(self):
        print("A loud cheer erupts from the crowd")

    # Example 2. Returning function as a value from other functions
    def ball_pass(self):
        '''A method which creates a new teammate and returns their score method'''
        return Player('teammate').score

# Create Ronaldo
ronaldo = Player('ronaldo')
> ronaldo steps onto the pitch

# Example 1. Passing functions as arguments to other functions
def play(action):
    '''Method which takes in a function and executes it'''
    action()
# Pass in ronaldo's kick function to the play function. The play function executes ronaldo's kick method.
play(ronaldo.kick)
> The ball went flying over the top of the goal

# 3. Assigning functions to variables
# Assign Ronaldo's ball pass method to the next_action variable
next_action = ronaldo.ball_pass()
> teammate steps onto the pitch
# Execute Ronaldo's ball pass method, which is now referenced by next_action
next_action()
> A loud cheer erupts from the crowd
# The same thing can be achieved by passing the ball pass method to the play method (similar to Example 2)
play(next_action)
> A loud cheer erupts from the crowd

As can be seen above, a function may be referenced using its name, and thus passed around from within and without other functions without being executed. Only when it is called - using () is the function code executed.

1. What is a decorator?

A decorator is simply syntactic sugar (i.e an easy notation) to execute code that wraps around a method.

Let’s start off with the most basic version of a retry method:

def retry(func):
    '''Receives a function, executes it and retries the execution if the return value is False'''
    max_attempts = 3
    attempts = 0
    while attempts <= max_attempts:
        print("Running %s, attempt %s" % (func.__name__, attempts))
        function_result = func()
        if function_result:
            return function_result
        attempts += 1

If we want to use this, we need to pass in the function we wish to retry as an argument:

def always_fail():
    return False

def always_succeed():
    return True

assert None == retry(always_fail)
> Running always_fail, attempt 1
> Running always_fail, attempt 2
> Running always_fail, attempt 3

assert True == retry(always_succeed)
> Running always_succeed, attempt 0

A decorator is simply an easy way of calling this wrapper function:

@retry
def always_fail():
    return False

@retry
def always_succeed():
    return True

If you ran the above, you would have seen the following output:

> Running always_fail, attempt 1
> Running always_fail, attempt 2
> Running always_fail, attempt 3
> Running always_succeed, attempt 0

Hang on a minute, the functions are being executed without even being called!

This is because the retry method executes the wrapped function directly for us, which is often not what we want. We want the ability to execute the wrapped function at our leisure, not when it is defined. In order to do so, the decorator must return a method rather than the executed result. That is, we need a wrapper within a wrapper, also known as a closure:

def retry(func):
    '''Receives a function, executes it and retries the execution if the return value is False'''
    def execute_func():
        max_attempts = 3
        attempts = 0
        while attempts <= max_attempts:
            print("Running %s, attempt %s" % (func.__name__, attempts))
            function_result = func()
            if function_result:
                return function_result
            attempts += 1
    return execute_func

Now, let’s see what our functions return when they are defined:

@retry
def always_fail():
    return False

@retry
def always_succeed():
    return True

print(always_fail)
> <function retry.<locals>.run_func at 0x03AC7B70>
print(always_succeed)
> <function retry.<locals>.run_func at 0x03AC7C00>

This seems more promising. The functions are not executed, instead we receive a reference to the run_func method, which will execute the wrapped function for us when called.

1.1 Preserving function names

However, it’s still not perfect. It would be preferable if the name of the function being called was preserved. After all, always_fail is more useful when debugging, than run_func, even in our contrived example, as the former is more specific.

We can use the functools.wraps decorator on the inner method of the decorator to preserve the original function name:

from functools import wraps
def retry(func):
    '''Receives a function, executes it and retries the execution if the return value is False'''
    @wraps(func)
    def run_func():
        max_attempts = 3
        attempts = 0
        while attempts <= max_attempts:
            print("Running %s, attempt %s" % (func.__name__, attempts))
            function_result = func()
            if function_result:
                return function_result
            attempts += 1
    return run_func

@retry
def always_fail():
    return False

@retry
def always_succeed():
    return True

print(always_fail)
<function always_fail at 0x032A7B28>
print(always_succeed)
<function always_succeed at 0x032A7BB8>

2. Decorator Classes

Often, it is desirable to define a decorator as a class. In python, any object can be a function if it implements the __call__ method. This is what makes an object callable. Let’s rewrite our retry method using a class:

class Retry():
    def __init__(self, func):
        print("Decorator has initialised, with function %s" % func.__name__)
        self.func = func
        self.max_attempts = 3

    def __call__(self):
        attempts = 0
        while attempts <= self.max_attempts:
            print("Running %s, attempt %s" % (self.func.__name__, attempts))
            function_result = self.func()
            if function_result:
                return function_result
            attempts += 1

There are a few notable differences between this class decorator and the function decorator:

Naively, this is how the class may be used without the decorator syntax:

# Decorator initialisation
retry = Retry(always_fail)
> Decorator has initialised, with function always_fail
# Decorator execution
retry()
> Running always_fail, attempt 0
> Running always_fail, attempt 1
> Running always_fail, attempt 2
> Running always_fail, attempt 3

And now with decorator syntax:

@Retry
def always_fail():
    return False
> Decorator has initialised, with function always_fail
always_fail()
> Running always_fail, attempt 0
> Running always_fail, attempt 1
> Running always_fail, attempt 2
> Running always_fail, attempt 3

3. Wrapping functions with arguments

In our examples so far we haven’t really dealt with functions with arguments. What if the function we were wrapping had arguments?

@Retry
def always_succeed(value1, value2):
    print("Running always succeed with value1: %s and value2: %s" % (value1, value2))
    return True

As we have learnt, when this method is executed, the __call__ method of Retry is executed, so we just need to make sure that the __call__ method accepts any arguments a wrapped function may have, and passes it on when it executes the wrapped function:

class Retry():
    def __init__(self, func):
        print("Decorator has initialised, with function %s" % func.__name__)
        self.func = func
        self.max_attempts = 3

    # Note: Add *args and **kwargs to accept any keyword arguments and positional arguments the wrapped function may have.
    def __call__(self, *args, **kwargs):
        attempts = 0
        while attempts <= self.max_attempts:
            print("Running %s, attempt %s" % (self.func.__name__, attempts))
            function_result = self.func(*args, **kwargs)
            if function_result:
                return function_result
            attempts += 1

always_succeed(1, 3)
> Running always_succeed, attempt 0
> Running always succeed with value1: 1 and value2: 3
always_succeed(5, 4)
> Running always_succeed, attempt 0
> Running always succeed with value1: 5 and value2: 4

4. Handling optional arguments

That was relatively painless! What happens if we wanted to create a decorator that is customisable using optional arguments. Note here that we’re not talking about arguments in the wrapped function, but the decorator itself.

This would be useful if say we wanted to customise the max number of attempts, or add a variable wait time between retries for our Retry decorator.

I’m sick of looking at the Retry class, so let’s take a step back and have a look at an even simpler decorator class to see how optional decorator arguments work:

class Kiss():
    def __init__(self, *args, **kwargs):
        print("Leaning in")
        print(args)
    def __call__(self, *args, **kwargs):
        print("Kissing")
        print(args)

@Kiss(tongue=False, gentle=True)
def wake_up(person="Imaginary Friend"):
    print("%s is waking up!" % person)

If you run the above snippet, you will receive the following output:

Leaning in
()
Kissing
(<function wake_up at 0x038D9DB0>,)

It looks like we’re back to Section 1, where the wrapped function is being called even though we didn’t explicitly call it! What’s happening here?

If we naively translate what the decorator is doing, it looks something like this:

# 1. Instantiate the kiss instance
kiss_instance = Kiss(tongue=False, gentle=True)
# 2. Use the kiss instance to initialise the decorator
kiss_instance(wake_up)

The key to understanding what the decorator is doing is to note that until now we have been using the decorator without calling it. But once we start using arguments, we explicitly call the decorator as well. This is the difference between using Kiss as @Kiss vs. @Kiss() or @Kiss(tongue=False, gentle=True). This comes back to what we were discussing in Section 0 - The Basics: python functions are first class functions, and this applies to decorators as well.

Now, going back to our Retry class - how can we make it handle being used with optional arguments as well as no arguments? i.e being used as @Retry, as well as @Retry(max_attempts=2, wait_time=20, backoff=3), or even @Retry().

The consequence of using arguments is two-fold:

  1. The wrapped function is passed in __call__ rather than in __init__

  2. __init__ and __call__ are both executed as soon as the decorator class is declared for a function.

To handle arguments as well as no arguments in the decorator, we need to know when the decorator is being called explicitly (i.e with arguments or empty arguments), and when it is simply defined with no arguments.

A possible solution is to check if the first argument of both __init__ and __call__ is a callable to determine where the wrapped function is being passed in, and therefore deduce how the decorator is being called. For eg. if the first argument of __init__ is a callable, then it is likely the decorator is being called without any arguments, i.e as @Retry, and alternatively if the first argument of __call__ is a callable, then the decorator was called with additional arguments - @Retry(wait_time=2), or explicitly with no arguments - @Retry().

Additionally, if the decorator has been called with arguments, we can return a wrapper which executes retries of the wrapped function from __call__ to delay execution of retry until the wrapped function is explicitly called. And if not, we can execute the wrapper within __call__, rather than returning it.

Here is our modified Retry:

class Retry():
    def __init__(self, *args, **kwargs):
        self.func = None
        self.max_attempts = kwargs.get('max_attempts', 3)
        self.wait_time = kwargs.get('wait_time', 30)
        self.backoff = kwargs.get('backoff', 2)

        # CASE 1: No Decorator Arguments
        if len(args) > 0 and callable(args[0]):
            self.func = args[0]
            print("Decorator has initialised with no arguments, with function %s" % self.func.__name__)
        print("max attempts: %s, wait time: %s, backoff: %s" % (self.max_attempts, self.wait_time, self.backoff) )

    # Note: Add *args and **kwargs to accept any keyword arguments
    # and positional arguments the wrapped function may have.
    def __call__(self, *args, **kwargs):
        def run_retry(*args, **kwargs):
            attempts = 0
            while attempts <= self.max_attempts:
                print("Running %s, attempt %s" % (self.func.__name__, attempts))
                function_result = self.func(*args, **kwargs)
                if function_result:
                    return function_result
                attempts += 1

        # CASE 2: Additional Decorator Arguments
        if len(args) > 0 and callable(args[0]):
            self.func = args[0]
            print("Decorator called using additional arguments, returning wrapper")
            return run_retry

        # CASE 1: No Decorator Arguments
        return run_retry(*args, **kwargs)

Its important to note that when we detect additional decorator arguments in __call__ we save the wrapped function to self.func. This is because the wrapper (run_retry) blindly executes retry on the function stored in self.func, when the wrapped function is called.

Lets see if this decorator behaves as expected when used with all of the options discussed so far:

@Retry
def always_succeed_1():
    print("1: Running function with no decorator args or function args ")
    return True

> Decorator has initialised with no arguments, with function always_succeed_1
> max attempts: 3, wait time: 30, backoff: 2

@Retry(max_attempts=3, backoff=2)
def always_succeed_2():
    print("2: Running function with decorator args and no function args")
    return True

> max attempts: 3, wait time: 30, backoff: 2
> Decorator called using additional arguments, returning wrapper

@Retry(max_attempts=2, backoff=4)
def always_succeed_3(live, live_again):
    print("3: Running function with decorator args and function args")
    return live

> max attempts: 2, wait time: 30, backoff: 4
> Decorator called using additional arguments, returning wrapper

always_succeed_1()
> Running always_succeed_1, attempt 0
> 1: Running function with no decorator args or function args

always_succeed_2()
> Running always_succeed_2, attempt 0
> 2: Running function with decorator args and no function args

always_succeed_3(True, True)
> Running always_succeed_3, attempt 0
> 3: Running function with decorator args and function args

As you can see, we have successfully created a flexible decorator which can be customised with optional arguments, or used “as is” without specifying any arguments.

And now finally, lets make use of those optional arguments to make Retry a little more useful. Below is the updated wrapper function in __call__ (run_retry) which uses the additional decorator variables to change the time waited between retry attempts defined by wait_time = attempt number * backoff * wait_time:

import time
...
        def run_retry(*args, **kwargs):
            wait_time, backoff, attempts = self.wait_time, self.backoff, 0
            while attempts < self.max_attempts:
                print("Running %s, attempt %s" % (self.func.__name__, attempts))
                function_result = self.func(*args, **kwargs)
                if function_result:
                    return function_result
                print("Waiting %s seconds before next attempt" % wait_time)
                time.sleep(self.wait_time)
                wait_time *= backoff
                attempts += 1

Lets see how this final Retry decorator handles failures:

@Retry(max_attempts=2, wait_time=10, backoff=2)
def always_fail(die):
    print("4: Running function with decorator args and function args")
    return die
> max attempts: 2, wait time: 10, backoff: 2
> Decorator called using additional arguments, returning wrapper

always_fail(False)
> Running always_fail, attempt 0
> 4: Running function with decorator args and function args
> Waiting 10 seconds before next attempt
> Running always_fail, attempt 1
> 4: Running function with decorator args and function args
> Waiting 20 seconds before next attempt

And that brings us to the end of our decorator adventures. There is a LOT more stuff you can do with decorators, and we’ve only really covered one useful example. If you want to learn more, I would suggest going through the decorator library, and this blog post.

Happy Decorating!