I was watching a really interesting video about buildings architecture in New York, when one of my DIT student asked me to explain to him what was happening in this piece of code. A function I have written few months ago to check for valid authentication tokens. In this post, I will try to show how decorators in Python, a powerful concept, allows us to apply new functionality without modifying the existing structure of an object. Music please.
TOC:
- 1 A function doing one thing
- 2 A decorator, modifying the behaviour of the function
- 3 A decorator should keep important informations
- 4 A decorator should receive a flexible number of argument
- 5 A decorator can also have parameters
- 6 We are still able to access the decorated function
- 7 An interesting example
- 8 Even more interesting, how decorators are used elsewhere
1. A function:
Let's suppose we want to join 2 paths:
In [2]: from pathlib import Path
In [3]: a = Path("foo")
In [4]: a
Out[4]: PosixPath('foo')
In [5]: b = Path("bar")
In [6]: a.joinpath(b)
Out[6]: PosixPath('foo/bar')
In [7]:
Let's suppose we usually combine 2 paths together. So much that we created a function for that purpose:
In [21]: path_t = str | Path
In [22]: def jp(path1: path_t, path2: path_t) -> path_t:
...: """Join 2 paths together"""
...: if isinstance(path1, str):
...: path1 = Path(path1)
...: return path1.joinpath(path2)
...:
In [23]: jp(b, a)
Out[23]: PosixPath('bar/foo')
In [24]: jp(a, b)
Out[24]: PosixPath('foo/bar')
In [25]: jp("corge", a)
Out[25]: PosixPath('corge/foo')
In [26]: jp("corge", "qux")
Out[26]: PosixPath('corge/qux')
2. A decorator modifying the behaviour of the function:
While it may seem confusing at first, a decorator is really just a function that takes a function and returns a function. Let me explain.
Suppose we usually combine our home path with another path. Let's suppose the first parameter is often our home path. Let's suppose we can't modify the original function. Python functions are first-class citizens. That means:
- We can pass a function as an argument of another function.
- We can assign a function to a variable.
- We can return a function as a value from another function.
In [38]: def homeprefix(func: Callable[[path_t, path_t], path_t]) -> Callable[[path_t], path_t]:
...: def wrapper(m: path_t) -> path_t:
...: """A wrapper that takes one parameter."""
...: return func("/home/nsukami", m)
...: return wrapper
...:
In [39]: f = homeprefix(jp)
In [40]: f("bar")
Out[40]: PosixPath('/home/nsukami/bar')
In [41]: f(b)
Out[41]: PosixPath('/home/nsukami/bar')
In [42]: f(a)
Out[42]: PosixPath('/home/nsukami/foo')
In [43]: f("corge/qux")
Out[43]: PosixPath('/home/nsukami/corge/qux')
In [44]: f(Path("spam"))
Out[44]: PosixPath('/home/nsukami/spam')
In [45]:
As strange as it may seems, we could have done the following:
In [45]: jp = homeprefix(jp) # there is an alternative syntax
In [46]: jp(a)
Out[46]: PosixPath('/home/nsukami/foo')
In [47]: jp(b)
Out[47]: PosixPath('/home/nsukami/bar')
In [48]: jp("foobar")
Out[48]: PosixPath('/home/nsukami/foobar')
In [49]: jp(Path("corge"))
Out[49]: PosixPath('/home/nsukami/corge')
In [50]:
Instead of jp = homeprefix(jp) we could use the following syntax which better vehicle our intent: modifying the behaviour of an existing function. Simply put: decorating our original function:
In [50]: def homeprefix(func):
...: def wrapper(m):
...: """A function that takes one parameter."""
...: return func("/home/nsukami", m)
...: return wrapper
...:
In [51]: @homeprefix # here we are, decorating jp function
...: def jp(path1: path_t, path2: path_t) -> path_t:
...: """Join 2 paths together"""
...: if isinstance(path1, str):
...: path1 = Path(path1)
...: return path1.joinpath(path2)
...:
In [52]: jp(Path("corge"))
Out[52]: PosixPath('/home/nsukami/corge')
In [53]: jp("corge/qux")
Out[53]: PosixPath('/home/nsukami/corge/qux')
In [54]: jp(a)
Out[54]: PosixPath('/home/nsukami/foo')
In [55]: jp(b)
Out[55]: PosixPath('/home/nsukami/bar')
In [56]:
3. A decorator should keep important informations:
Now we have another problem. Because we are literally replacing one function with another, we're also losing important informations like the docstring and the name:
In [61]: ?jp
Signature: jp(m)
Docstring: A function that takes one parameter.
File: ~/GIT/nskm2/<ipython-input-59-b4f67b1e800d>
Type: function
In [62]: jp.__name__
Out[62]: 'wrapper'
In [63]:
That's the reason we have functools.wraps. Wraps takes the decorated function and adds the functionality of copying over the function name, docstring, arguments list, etc to the returned function. And since wraps is itself a decorator:
In [63]: from functools import wraps
In [64]: def homeprefix(func):
...: @wraps(func)
...: def wrapper(m):
...: """A function that takes one parameter."""
...: return func("/home/nsukami", m)
...: return wrapper
...:
In [65]: @homeprefix
...: def jp(path1: path_t, path2: path_t) -> path_t:
...: """A function that join 2 paths together."""
...: if isinstance(path1, str):
...: path1 = Path(path1)
...: return path1.joinpath(path2)
...:
In [66]: ?jp
Signature: jp(path1: str | pathlib.Path, path2: str | pathlib.Path) -> str | pathlib.Path
Docstring: A function that join 2 paths together.
File: ~/GIT/nskm2/<ipython-input-65-1703cd8a40dc>
Type: function
In [67]: jp.__name__
Out[67]: 'jp'
In [68]:
4. A decorator should receive a flexible number of argument:
Now, let's suppose our original function can take many arguments:
In [101]: def jp(*args: path_t) -> path_t:
...: """A function that join many paths together."""
...: if args:
...: path1 = Path(args[0])
...: return path1.joinpath(*args[1:])
...: else:
...: raise ValueError("args should be longer")
...:
In [102]: jp("bac", "abc")
Out[102]: PosixPath('bac/abc')
In [103]: jp("bac")
Out[103]: PosixPath('bac')
In [104]: jp(Path("abc"), Path("foo"), "bar")
Out[104]: PosixPath('abc/foo/bar')
The previous decorator is not flexible enough because the returned function can't take more than 1 parameter:
In [106]: def homeprefix(func):
...: @wraps(func)
...: def wrapper(m):
...: """A function that takes one parameter."""
...: return func("/home/nsukami", m)
...: return wrapper
...:
In [107]: @homeprefix
...: def jp(*args: path_t) -> path_t:
...: """A function that join many paths together."""
...: if args:
...: path1 = Path(args[0])
...: return path1.joinpath(*args[1:])
...: else:
...: raise ValueError("args should be longer")
...:
...:
In [108]: jp("a")
Out[108]: PosixPath('/home/nsukami/a')
In [109]: jp("a", "b")
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [109], line 1
----> 1 jp("a", "b")
TypeError: jp() takes 1 positional argument but 2 were given
In [110]:
Let's rewrite our decorator so that the returned function can take an infinte number or arguments:
In [113]: def homeprefix(func):
...: @wraps(func)
...: def wrapper(*args): # here is our update: using *args
...: return func("/home/nsukami", *args)
...: return wrapper
...:
In [114]: @homeprefix
...: def jp(*args: path_t) -> path_t:
...: """A function that join many paths together."""
...: if args:
...: path1 = Path(args[0])
...: return path1.joinpath(*args[1:])
...: else:
...: raise ValueError("args should be longer")
...:
In [115]: jp()
Out[115]: PosixPath('/home/nsukami')
In [116]: jp("b", "c", "d")
Out[116]: PosixPath('/home/nsukami/b/c/d')
In [117]: jp("b", "c", "d", Path("foo"))
Out[117]: PosixPath('/home/nsukami/b/c/d/foo')
In [118]: jp(Path("foo"), Path("bar"))
Out[118]: PosixPath('/home/nsukami/foo/bar')
In [119]:
With the help of the special parameters *args and **kwargs, decorators become more generic. That way if we need the same kind of decorator in different functions with different arguments, we can write just one decorator. In general, a decorator is written like this:
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
func(*args, **kwargs)
return wrapper
5. A decorator can also have parameters:
Let's suppose people using the decorator want to be able to set what should the prefix used as the first parameter of our jp function. Let's suppose we want the decorator to receive one parameter, which is the prefix:
In [70]: def homeprefix(prefix): # this function builds and returns a decorator
...: def outer(func): # this function is the decorator
...: @wraps(func)
...: def inner(*args, **kwargs): # the function wrapping the original function
...: """A function that takes one parameter."""
...: return func(prefix, *args, **kwargs)
...: return inner
...: return outer
...:
In [71]: @homeprefix("/home/nsukami")
...: def jp(path1: path_t, path2: path_t) -> path_t:
...: """A function that join 2 paths together."""
...: if isinstance(path1, str):
...: path1 = Path(path1)
...: return path1.joinpath(path2)
...:
In [72]: jp(a)
Out[72]: PosixPath('/home/nsukami/foo')
In [73]: jp(b)
Out[73]: PosixPath('/home/nsukami/bar')
In [74]: jp("corge")
Out[74]: PosixPath('/home/nsukami/corge')
In [75]:
We understand better what's going on when we do this instead:
In [75]: decorator = homeprefix("/home/patrick") # setup a prefix
In [76]: @decorator # decorate
...: def jp(path1: path_t, path2: path_t) -> path_t:
...: """A function that join 2 paths together."""
...: if isinstance(path1, str):
...: path1 = Path(path1)
...: return path1.joinpath(path2)
...:
In [77]: jp("corge")
Out[77]: PosixPath('/home/patrick/corge')
In [78]: jp("quux")
Out[78]: PosixPath('/home/patrick/quux')
In [79]:
6. We are still able to access the decorated function:
Let's suppose for some reason we want to test the function as if it was not decorated. Let's suppose we still want to use the underlying function. To achieve that, we'll use the __wrapped__ attribute (assuming we used the functools.wrap function). Somewhere in the documentation, we can read:
The original underlying function is accessible through the wrapped attribute. This is useful for introspection, for bypassing the cache, or for rewrapping the function with a different cache.
In [127]: jp("foo")
Out[127]: PosixPath('/home/nsukami/foo')
In [128]: jp.__wrapped__("foo") # original function
Out[128]: PosixPath('foo')
In [129]:
7. An interesting example:
#!/usr/bin/env python3
import csv
import sqlite3
import pathlib
from functools import wraps
here = pathlib.Path(__file__)
db = here.parent.parent.parent / "sqlite_stuff" / "chinook" / "chinook.db"
def connect(db):
"""A decorator to connect to an SQLite database and execute a query."""
conn = sqlite3.connect(db)
conn.isolation_level = None
def outer(f):
@wraps(f)
def inner(*args, **kwargs):
query = f(*args, **kwargs)
rset = conn.execute(query).fetchall()
return rset
return inner
return outer
def columns(*columns, limit=10):
"""A decorator to specify what columns should be retrieved."""
cols = ", ".join(columns)
def outer(f):
@wraps(f)
def inner(*args, **kwargs):
query = f(*args, **kwargs).split("*")
start, end = query[0].strip(), query[1].strip()
return f"{start} {cols} {end} limit {limit}"
return inner
return outer
def as_csv(where="./result.csv"):
"""A decorator to save the result as a csv file."""
def outer(f):
@wraps(f)
def inner(*args, **kwargs):
rset = f(*args, **kwargs)
with open(where, "w", newline="") as dest:
csv_writer = csv.writer(
dest, delimiter=",", quoting=csv.QUOTE_ALL, lineterminator=";\r\n"
)
csv_writer.writerows(rset)
return len(rset)
return inner
return outer
###
@as_csv()
@connect(db)
@columns("title", "artistId")
def run(table="albums"):
query = f"select * from {table}"
return query
if __name__ == "__main__":
print(run())
8. Even more interesting:
import sqlite3
import pathlib
from functools import wraps
here = pathlib.Path(__file__)
db = here.parent.parent.parent / "sqlite_stuff" / "chinook" / "chinook.db"
def connect(cls):
conn = sqlite3.connect(db)
conn.isolation_level = None # sqlite3.IMMEDIATE
setattr(cls, "conn", conn)
return cls
def select(cls):
def select(self):
return cls.conn.execute(f"select * from {cls.__name__}").fetchall()
setattr(cls, "select", select)
return cls
@select
@connect
class Albums:...
if __name__ == "__main__":
a = Albums()
print(a.select())
8. Even more interesting, how decorators are used elsewhere:
- Cubicweb decorator used to raise an AuthenticationError if no user is detected.
- Flask decorator that redirects anonymous users to the login page.
- Templating decorator invented by the TurboGears guys a while back to automatically render a template.
- Django decorators to manipulate http headers.
9. More on this topic:
I hope that you found this post helpful. To learn about this topic, please, browse the following links: