Quick note

This article is a work in progress. And there’s lots of refereces. Don’t try to read all of them, at least not in one go. It’s too much info. Just focus on what’s relevant for you right now and come back later when you need more info. At the end there’s a reference section with all the links, there could even be links there that are not in the article.

Introduction

We’ll see several ways of debugging in Python. From easy print debugging to pdb.
Most of the time we want something better than print but don’t really need a full fledged debugger. For that kind of situation, we’ll use pysnooper. When we need a real debugger, we’ll use pdb, or one of its derivatives.

I’m not going to cover IDE debugger for now (todo). The reason is that depending of your IDE, execution platform, OS, etc. The setup can be quite different, and in some scenarios complicated. Btw, by execution platform I mean in which machine the code is executed (local, remote), and whether it’s running directly on the OS or in a container, VM, etc. PDB will always work and it’s easy to setup.

Python debugging methods/tools

Easy, dirty Debugging with `print`

Everybody know this technique and uses it on a daily basis. It’s easy, it’s quick, it’s dirty. With this method, sometimes you need to write a lot, use some pseudo-unique string at the start of each print in order to know were you are in the code, etc… 🥴 Oh, and did you ever commit 234342 lines of print accidentally? I did. Everyone does. What’s worse, often you add a few print, then you run the code, and realice you need to print another variable. Or you may know you need more info, but you’re not sure what. So you add a few print, run the code, and repeat. This makes you waste a lot of time in that add print, run, add print, run, … loop. Sometimes it is just the best option. Sometimes, not so much.

Easy, nice Debugging with `pysnooper`

pysnooper is a great tool for debugging. It’s easy to use, and it’s very powerful. 💪 You can think of it as a print on steroids. It provides a function -actually a class, but who cares- that you can use as a decorator, or as a context manager to either debug a function or a block of code. It’s so easy to use, that I’m not going to really explain anything, I’ll just show you a few examples borrowed from the pysnooper README.

Examples

Example

We’re writing a function that converts a number to binary, by returning a list of bits. Let’s snoop on it by adding the @pysnooper.snoop() decorator:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
import pysnooper

@pysnooper.snoop()
def number_to_bits(number):
    if number:
        bits = []
        while number:
            number, remainder = divmod(number, 2)
            bits.insert(0, remainder)
        return bits
    else:
        return [0]

number_to_bits(6)

The output to stderr is:

Or if you don’t want to trace an entire function, you can wrap the relevant part in a with block:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
import pysnooper
import random

def foo():
    lst = []
    for i in range(10):
        lst.append(random.randrange(1, 1000))

    with pysnooper.snoop():
        lower = min(lst)
        upper = max(lst)
        mid = (lower + upper) / 2
        print(lower, mid, upper)

foo()

which outputs something like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
New var:....... i = 9
New var:....... lst = [681, 267, 74, 832, 284, 678, ...]
09:37:35.881721 line        10         lower = min(lst)
New var:....... lower = 74
09:37:35.882137 line        11         upper = max(lst)
New var:....... upper = 832
09:37:35.882304 line        12         mid = (lower + upper) / 2
74 453.0 832
New var:....... mid = 453.0
09:37:35.882486 line        13         print(lower, mid, upper)
Elapsed time: 00:00:00.000344

Features

If stderr is not easily accessible for you, you can redirect the output to a file:

1
@pysnooper.snoop('/my/log/file.log')

You can also pass a stream or a callable instead, and they’ll be used.

See values of some expressions that aren’t local variables:

1
@pysnooper.snoop(watch=('foo.bar', 'self.x["whatever"]'))

Show snoop lines for functions that your function calls:

1
@pysnooper.snoop(depth=2)

I highly recommend reading the whole pysnooper README, and the advanced usage page.
You could also check out this talk by the author of pysnooper.

Proper debugging with `pdb`

There’s lots of time when a few print or pysnooper.snoop are not enough.
PDB got you.
The most basic usage is to add a import pdb; pdb.set_trace() line in your code. This is equivalent to a breakpoint. And in fact, since Python 3.7, thanks to PEP 553 you can use breakpoint() instead of import pdb; pdb.set_trace().
I recommend using breakpoint(), because you can use env vars to configure what debugger to use, whether to actually stop or not. If you run your code with PYTHONBREAKPOINT=0 python mycode.py, it won’t stop at any breakpoint.
If you have ipdb installed, you can use PYTHONBREAKPOINT=ipdb.set_trace python mycode.py to use ipdb instead of pdb, so that you can use IPython’s debugger, with tab completion, syntax highlighting, and a IPython shell when you use the interact command, instead of a plain Python shell. You can learn more about ipdb in the ipdb docs.

Good resources to learn about pdb are Python Docs PDB, Python Docs Breakpoint and this two videos of anthonywritescode: breakpoint and pdb.

Assuming you have access to a terminal, if you want to debug in a docker container, or in a remote machine, you can just use PDB or any of its derivatives. In case you don’t have access to a terminal, you can use Remote PDB. And if you want a debugger that implements the Debug Adapter Protocol, for example to easily use youe neovim debugger in a container/remote machine, you can use Debugpy. And since it is a protocol, any debugger that implements it, can be used. For example, VS Code debugger uses it too.

One think you should have in mind in case you want yo use pdb in a docker compose service, is that you need yo add a few thinks to your config, in order to get an interactive terminal. By default it is not.

Lets say you have a compose file like this:

1
2
3
4
5
services:
  django: &django
    build:
      context: .
      dockerfile: ./compose/local/django/Dockerfile

To make the terminal interactive, add tty: true and stdin_open: true to the service config:

1
2
3
4
5
6
7
services:
  django: &django
    build:
      context: .
      dockerfile: ./compose/local/django/Dockerfile
    stdin_open: true
    tty: true

Now you can run docker-compose up --attach django and you’ll get an interactive terminal.

If you want to make sure you don’t commit your code with any breakpoint() or import pdb; pdb.set_trace() left, you can use a pre-commit hook to check for them. If you are using Pre Commit, you can use the debug-statements hook. That’s nice because there’s some prints you want to preserve, so it doesn’t make sense to use a hook to check there’s no print left. But you likely don’t want to commit any breakpoint().
.pre-commit-config.yaml:

1
2
3
4
5
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0
    hooks:
      - id: debug-statements

Guide of Python Debugging w/ tips & tricks (wip)

Quick note

Introduction

Python debugging methods/tools

Easy, dirty Debugging with `print`

Easy, nice Debugging with `pysnooper`

Example

Features

Proper debugging with `pdb`

References

Quick note#

Introduction#

Python debugging methods/tools#

Easy, dirty Debugging with print#

Easy, nice Debugging with pysnooper#

Example#

Features#

Proper debugging with pdb#

References#

Quick note

Introduction

Python debugging methods/tools

Easy, dirty Debugging with `print`

Easy, nice Debugging with `pysnooper`

Example

Features

Proper debugging with `pdb`

References