Using typing.Generic in Python

I am working on a project with a relatively large code-base. And it has a history. Our team wrote some parts of it way before type-hints. And we steel add hints to the legacy code or improve them. Is it worth the fuss? Sure! Our users are developers. They open our code in PyCharm every day. And they hope that it will help them to solve their task as quickly as possible.

I can say that there is a correlation between how precise code completion is and how fast developers work. However, our goal is not only to introduce more precise static analysis and code completion in PyCharm but not to break their production code.

"not to break their production code" means that we can't change class names and their API if it is not necessary. And type-hint is not that case.

One of my colleagues added a hint to the code that looks like this:

# file name: type_cast_0.py
class A:
    a = 'a'


class B(A):
    b = 'b'


class DoSomethingWithA:
    _class = A

    def do(self) -> A:
        return self._class()


class DoSomethingWithB(DoSomethingWithA):
    _class = B

PyCharm is OK with that code. Its analyzer shows a green checkmark. Mypy doesn't show any errors either:

$ mypy type_cast_0.py
Success: no issues found in 1 source file

Let’s add a code that uses DoSomethingWithB.

# file name: type_cast_1.py
from type_cast_0 import DoSomethingWithB

print(DoSomethingWithB().do().b)

PyCharm shows a warning immediately: Unresolved attribute reference 'b' for class 'A'. Mypy also marks it as an error.

$ mypy type_cast_1.py
type_cast_1.py:4: error: "A" has no attribute "b"
Found 1 error in 1 file (checked 1 source file)

Let’s fix it. Here is my first approach. It is a naive use of generics in Python.

# file name: type_cast_2.py
#...
TV = tp.TypeVar('TV')


class DoSomethingWithA(tp.Generic[TV]):
    _class: tp.Type[TV] = A

    def do(self) -> TV:
        return self._class()


class DoSomethingWithB(DoSomethingWithA):
    _class = B

PyCharm shows no errors or warnings. But mypy does.

$ mypy type_cast_3.py
type_cast_2.py:17: error: Incompatible types in assignment (expression has type "Type[A]", variable has type "Type[TV]")
Found 1 error in 1 file (checked 1 source file)

Interesting… Trying to change TV = tp.TypeVar('TV') to TV = tp.TypeVar('TV’, bound=A). Same error. More interesting…

Official python documentation isn’t helpful in this case. There are a couple of examples of using Generics, but nothing that gives an idea of how to fix the issue. Hopefully, there is a brilliant section about Generics on the mypy website.

For my example, the resulting code would be like this one.

# file name: type_cast_6.py
# ...
class DoSomethingWith(tp.Generic[TV]):
    _class: tp.Type[TV]

    def do(self) -> TV:
        return self._class()

Here is a usage example.

# file name: type_cast_7.py
from type_cast_6 import DoSomethingWith, B

print(DoSomethingWith[B]().do().b)

Mypy shows no problems. Neither does PyCharm.

$ mypy type_cast_6.py
Success: no issues found in 1 source file
$ mypy type_cast_7.py
Success: no issues found in 1 source file

But actual usage of the code throws an exception.

$ python type_cast_7.py
...
AttributeError: 'DoSomethingWith' object has no attribute '_class'

Unfortunately, there is no way to use TypeVar as you can use generics in Java, for example. So I can’t assign TV to _class and expect from Python that it replace type variable to actual class during the execution. In other words, if I use _class: tp.Type[TV] = TV in type_cast_6.py, I’ll get an TypeError: 'TypeVar' object is not callable.

To avoid that I add subclasses for DoSomethingWithz.

# file name: type_cast_8.py
# ...
class DoSomethingWithA(DoSomethingWith):
    _class = A


class DoSomethingWithB(DoSomethingWith):
    _class = B

# file name: type_cast_9.py
from type_cast_8 import DoSomethingWithB

print(DoSomethingWithB().do().b)

Not particularly elegant, but it works.

This post has a lot of code examples. I truncated them a bit. You can find full versions of the examples on my GitHub account.