Variant

I was wandering on the internets, browsing Analogue Pocket reviews before buying it for myself. Then I decided to do something a little bit less productive: finish and publish this article. Part one, part two.


Context:

Discovering and playing with Python type annotations is really fun and interesting. Especially for someone who never really used them before like me.

Here is a perfectly harmeless Python function that return something depending on something:

def foo(x):
    if x == 1:
        return 42
    elif x == 2:
        return "bar"
    else:
        raise TypeError("Invalid argument")


If we’re asked to add type annotations to this function, we can do the following:

from typing import Union

def foo(x: int) -> Union[int, str]:
    if x == 1:
        return 42
    elif x == 2:
        return "bar"
    else:
        raise TypeError("Invalid argument")

We’re comparing x variable to the integers 1 & 2, so it sounds like our function will take an Integer as parameter. Depending on some conditions, we’re returning an Integer or a String. So, foo is a function that takes an Integer and returns either an Integer or a String. Mypy seems to be happy:

> mypy foo.py
Success: no issues found in 1 source file


What’s enthralling:

It is interesting to see that, from Mypy perspective, the revealed types for the variables a and b are exactly the same:

a = foo(1)  # We know a is an Integer
b = foo(2)  # We know b is a String

# Yet
reveal_locals()
> mypy foo.py  # Yet
foo.py:43: note: Revealed local types are:
foo.py:43: note:     a: Union[builtins.int, builtins.str]
foo.py:43: note:     b: Union[builtins.int, builtins.str]


Even more enthralling:

c: int
c = foo(1) # Forbidden, even if the returned value is indeed an Integer

foo(2) + "bar" # Forbidden, even if the returned value is indeed a String
> mypy foo.py  # Yet again
foo.py:65: error: Incompatible types in assignment (expression has type "Union[int, str]", variable has type "int")
foo.py:78: error: Unsupported operand types for + ("int" and "str")
foo.py:78: note: Left operand is of type "Union[int, str]"
Found 2 errors in 1 file (checked 1 source file)


Relationship, argument, return types:

There is a way to tell Mypy that there is a relationship between the argument and the type of the returned value. There is way to annotate the foo function so that reveal_locals will be able to more precisely tells us what are the types for a & b. Overloads to the rescue.

An overloaded function must consist of two or more overload variants followed by an implementation:

from typing import overload, Literal

@overload
def foo(x: Literal[1]) -> int: ...

@overload
def foo(x: Literal[2]) -> str: ...

def foo(x):
    if x == 1:
        return 42
    elif x == 2:
        return "bar"
    else:
        raise TypeError("invalid argument")
        
a = foo(1)
b = foo(2)
reveal_locals()
        
>> mypy foo.py
mixin5.py:105: note: Revealed local types are:
mixin5.py:105: note:     a: builtins.int
mixin5.py:105: note:     b: builtins.str
e: int
e = foo(1)

f: str
f = foo(2)

foo(1) + 1
foo(2) + "bar"
>> mypy foo.py
Success: no issues found in 1 source file


Can we have some real use cases?

We’ve seen a very simple use for variants, the following are some uses of them inside the stdlib.

The pow function:

From the documentation, we can read:

Help on built-in function pow in module builtins:

pow(base, exp, mod=None)
    Equivalent to base**exp with 2 arguments or base**exp % mod with 3 arguments
    
    Some types, such as ints, are able to use a more efficient algorithm when
    invoked using the three argument form.
(END)


Let’s play a little bit with this function:

  1. if the 3rd argument equals zero, don’t return:

    In [4]: pow(foo, 2, 0)  
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-4-9eeac4b049b8> in <module>
    ----> 1 pow(foo, 2, 0)
    
    ValueError: pow() 3rd argument cannot be 0
  2. if the 2nd argument is a positive Integer, return an Integer:

    In [5]: pow(foo, 2, 3)  
    Out[5]: 1
  3. if the second argument is a negative Integer, return a Float:

    In [6]: pow(foo, -2)
    Out[6]: 0.0625
    
    In [7]: 

The truth is, all the above witnessed behaviour are correctly documented here:

    @overload
    def __pow__(self, __x: int, __modulo: Literal[0]) -> NoReturn: ...
    @overload
    def __pow__(self, __x: int, __modulo: int) -> int: ...
    @overload
    def __pow__(self, __x: _PositiveInteger, __modulo: None = ...) -> int: ...
    @overload
    def __pow__(self, __x: _NegativeInteger, __modulo: None = ...) -> float: ...
    # positive x -> int; negative x -> float
    # return type must be Any as `int | float` causes too many false-positive errors
    @overload
    def __pow__(self, __x: int, __modulo: None = ...) -> Any: ...

The gcd function:

Before Python 3.9, the gcd function was defined inside the fractions module. The function was receiving 2 arguments. If one of those arguments was an Integral, the returned value was an Integral:

Python 2.7.18 (default, Sep 10 2021, 14:59:31) 
[GCC 11.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import fractions
>>> fractions.gcd(1, 2, 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: gcd() takes exactly 2 arguments (3 given)
>>> 
>>> 
>>> fractions.gcd(1, 2)
1
>>> 

The related type annotations:

if sys.version_info < (3, 9):
    @overload
    def gcd(a: int, b: int) -> int: ...
    @overload
    def gcd(a: Integral, b: int) -> Integral: ...
    @overload
    def gcd(a: int, b: Integral) -> Integral: ...
    @overload
    def gcd(a: Integral, b: Integral) -> Integral: ...

Since 3.9, it is defined inside the math module. Now, gcd takes only one argument, a sequence of Integers, and returns an Integer:

if sys.version_info >= (3, 9):
    def gcd(*integers: SupportsIndex) -> int: ...

else:
    def gcd(__x: SupportsIndex, __y: SupportsIndex) -> int: ...
mind blown

mind blown


Have you ever used overloads?

I’ve used them once, I was contributing some type annotations to the Typeshed repository, for the Dateparser package. Never used them during the last couple of months I spent adding type annotations to RQL.


Overload of informations on this topic:

Really hope you’ve learned somthing. In another article, I’ll try to show you a spellbinding use of overloads + decorators. Before that, let me suggest the following links for an in depth exploration:

  1. Function method overloading
  2. AnyOf - Union for return types
  3. Document why having a union return type is often a problem


overload in progress

overload in progress