Skip to content

Support to jump between chained exception in Pdb #106670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Carreau opened this issue Jul 12, 2023 · 6 comments · Fixed by #106676
Closed

Support to jump between chained exception in Pdb #106670

Carreau opened this issue Jul 12, 2023 · 6 comments · Fixed by #106676
Labels
stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@Carreau
Copy link
Contributor

Carreau commented Jul 12, 2023

Feature or enhancement

I believe it would be useful to allow to move between chained exception in Pdb.
That is to say if you have

try:
   function_that_raises()
except Exception as e:
   raise ...  from e  # Pdb says this is the bottom frame.

Havin something like down go down the chained exception ? This was requested on IPython's ipdb, But I believe this could be brought to Pdb as well.

Picth

When moving up and down the stack in Pdb in chained exception, when hitting the bottom of the stack, it should be possible to look at the __cause__ or __context__ and jump to the top of it. And vice-versa when going up if we store the various TB we went through.

In post-mortem debugging this should allow to better understand the reason for an exception.

Previous discussion

See,
https://discuss.python.org/t/interested-in-support-to-jump-between-chained-exception-in-pdb/29470/3

Linked PRs

@Carreau Carreau added the type-feature A feature request or enhancement label Jul 12, 2023
Carreau added a commit to Carreau/cpython that referenced this issue Jul 12, 2023
This lets Pdb receive and exception, instead of a traceback, and when
this is the case and the exception are chained, up/down allow to move
between the chained exceptions when reaching the tot/bottom.

That is to say if you have something like

    def out():
        try:
            middle()                                # B
        except Exception as e:
            raise ValueError("foo(): bar failed")   # A

    def middle():
        try:
            return inner(0)                         # D
        except Exception as e:
            raise ValueError("Middle fail")         # C

    def inner(x):
        1 / x                                       # E

Only A is reachable after calling `out()` and doing post mortem debug.
With this all A-E points are reachable with up/down.

I do not change the default behavior of ``pdb.pm()``, but I think that
arguably the default should be to pass `sys.last_value` so that chained
exception navigation is enabled.

Closes pythongh-106670
@gaogaotiantian
Copy link
Member

The general idea is great, yes giving the users the ability to take a look at chained exceptions is a good feature for pdb.

However, we probably need extra discussions for how this should be achieved.

  • How would the users use it? The current entry is too deep and no user would ever use pdb like that.
  • Is reusing up and down a good idea?
    • First of all, I think the current implementation is inverted. up in stack means "going backward in time" because it's going to the callers, which should be used to get the previous chained exception. That's how the current traceback of chained exception is displayed, and that's the ipdb issue suggested.
    • I'm having concerns for reusing up and down for this because it's not exactly the same thing. It's not completely irrelavent, but still.
    • If we decide to reuse up and down, we should probably give some hints to the users about it - it's kind of secluded now. If I were a debugger user, having a call stack with ALL the exceptions could be interesting - like the traceback. Of course we should have some indicators to seperate the exceptions.
    • However, this solution has its own problem, as the outermost exception could've had the full call stack - it's a bit weird to stepping up to the very top caller then to the previous exception.
    • The other solution could be having a set of new commands for chained exceptions to switch between them(is it an overkill?).

@Carreau
Copy link
Contributor Author

Carreau commented Jul 12, 2023

Thanks for the feedback,

  • How would the users use it? The current entry is too deep and no user would ever use pdb like that.

I want to decouple implementation from actually exposing it, I think in the end, ``import pdb; pdb.pm()` should just make this work.

First of all, I think the current implementation is inverted. up in stack means "going backward in time" because it's going to the callers, which should be used to get the previous chained exception. That's how the current traceback of chained exception is displayed, and that's the ipdb issue suggested.

I disagree with this – I also find the normal chained traceback backward.

Aparte

Instead of DeepException During handling of the above exception, another exception occurred: MiddleException ... I would prefer MiddleException Was cause by DeeperException.
I guess it depends wether the reraise was on purpose, which I guess is more explicit with raise ... from ..., or wether an actual error append during the error handling

End Aparte, back to original

I don't think we can use up to go to __cause__/__context__.
Say you have

 1 def out():
 2   try:
 3        i_fail()
 4   except:
 5        raise ValueError('failed')  # here.
 6
 7 def i_fail():
 8    1/0
 9 
10 out()

For me i_fail() is definitely a callee of out(), so moving from out() to i_fail() is going down. There is no notion of time in this.

And anyway say you are in the debugger at # here on line 5; up already brings you to out() on line 10, the only place you can go is therefore down... and you reach the top of i_fail() call.

I think that might be what you are trying to say with the following.

However, this solution has its own problem, as the outermost exception could've had the full call stack - it's a bit weird to stepping up to the very top caller then to the previous exception.

After playing a bit with it, I do find the current up/down pretty natural, if you think about it in term of intermediate try/except chained exception being absent. You just happen to stop on the except Exception as e: raise ... line. This is also highlighted by the fact that the original issue mentions Exception.add_note, so I believe we should really view this without the intermediate try/except.

The other solution could be having a set of new commands for chained exceptions to switch between them(is it an overkill?).

One thing that might settle this is exception groups, is we add commands to move between chained exception we can extend those to move between sub exception groups. I do not have async project where I use exception groups though so this is pure speculation.

Anyway, happy to go with the route cpython prefers.

@gaogaotiantian
Copy link
Member

Instead of DeepException During handling of the above exception, another exception occurred: MiddleException ... I would prefer MiddleException Was cause by DeeperException.
I guess it depends wether the reraise was on purpose, which I guess is more explicit with raise ... from ..., or wether an actual error append during the error handling

Unless you can convince core devs to change that in CPython, I'm afraid we should respect that order :)

For me i_fail() is definitely a callee of out(), so moving from out() to i_fail() is going down. There is no notion of time in this.

And anyway say you are in the debugger at # here on line 5; up already brings you to out() on line 10, the only place you can go is therefore down... and you reach the top of i_fail() call.

Only if your chained exception is raised in the same frame as your try ... except block. I guess your thoughts about the usages are more about a direct raise in the exception handling code, but what about:

def out():
    try:
        i_fail()
    except Exception as e:
        handle_exception(e)

def i_fail():
    1 / 0

def handle_exception(e):
    even_more_functions(e)

def even_more_functions(e):
    raise ValueError("Can't handle this")  # This is the exception raised

out()

How could down make any sense here?

I'm not saying up is a good choice, but it's conforming to the current traceback (that you did not like).

That's why I ask the question up front - is up and down good candidates for such feature? What we really have is multiple exceptions that are linked together as they were raised when handling each other.

I think a better solution might be a separate command to list and track exceptions, for example, exception. Like breakpoints, exception can list out all the current exceptions in post_mortem mode (even on exception event in normal debugging). exception <exc_number> will switch the context to that exception. We probably will be able to handle exception groups this way.

@Carreau
Copy link
Contributor Author

Carreau commented Jul 13, 2023

I'll try to implement that as a do_exception command.

Do you still think we should have interaction take an exception, or should be instead accept a list of tracebacks?

@gaogaotiantian
Copy link
Member

We want this to work for pm and post_mortem eventually (probably in this PR), but I doubt if we can simply change the input type for post_mortem to Exception - that's a breaking change. So we probably need to keep the traceback path anyway. I'd like @iritkatriel's input here as she's been reviewing my pdb code and she's the expert in Exceptions.

From my point of view, we need to support both Exception and traceback for post_mortem - traceback itself is not enough to get all the exception chain information. The questions is whether we add another argument for post_mortem (which would keep the full backward compatibility but the two arguments are kind of mutually exclusive), or we make the current argument which takes an exception (which is better code, but either need to use the word "traceback" for potential exception, or break backward compatibility for those who use the keyword "traceback" to call the function).

I know you asked about interaction, but the way we handle the user interface can be the way we handle interaction - however it's not an exposed API so we have a bit more flexibility there.

We definitely still need the path for traceback for interaction as we need to keep the path for post_mortem, but I don't like the idea of taking a list of tracebacks, as we'll lose the data for the exceptions. I think we need to store the exception(or exceptions) in pdb so we can display it.

Carreau added a commit to Carreau/cpython that referenced this issue Jul 18, 2023
This lets Pdb receive and exception, instead of a traceback, and when
this is the case and the exception are chained, `exceptions` allow to
list and move between the chained exceptions when reaching the
tot/bottom.

That is to say if you have something like

    def out():
        try:
            middle()                                # B
        except Exception as e:
            raise ValueError("foo(): bar failed")   # A

    def middle():
        try:
            return inner(0)                         # D
        except Exception as e:
            raise ValueError("Middle fail")         # C

    def inner(x):
        1 / x                                       # E

Only A is reachable after calling `out()` and doing post mortem debug.
With this all A-E points are reachable with a combination of up/down,
and ``exception <number>``.

I also change the default behavior of ``pdb.pm()``, to receive
`sys.last_value` so that chained exception navigation is enabled.

Closes pythongh-106670
@Carreau
Copy link
Contributor Author

Carreau commented Jul 18, 2023

I rebased/pushed and updated the description of #106676

  • pdb.pm(), now use last_value if present by default.
  • I kept the traceback name for parameters, and check with isinstance(..., BaseException).
  • I defined do_exceptions that takes:
    • nothing/"list": to enumerate the exception, and have a cursor to current exception.
    • a number: to jump to that exception.
    • The exception order is the same than a normal traceback

Carreau added a commit to Carreau/cpython that referenced this issue Jul 18, 2023
This lets Pdb receive and exception, instead of a traceback, and when
this is the case and the exception are chained, `exceptions` allow to
list and move between the chained exceptions when reaching the
tot/bottom.

That is to say if you have something like

    def out():
        try:
            middle()                                # B
        except Exception as e:
            raise ValueError("foo(): bar failed")   # A

    def middle():
        try:
            return inner(0)                         # D
        except Exception as e:
            raise ValueError("Middle fail")         # C

    def inner(x):
        1 / x                                       # E

Only A is reachable after calling `out()` and doing post mortem debug.
With this all A-E points are reachable with a combination of up/down,
and ``exception <number>``.

I also change the default behavior of ``pdb.pm()``, to receive
`sys.last_value` so that chained exception navigation is enabled.

Closes pythongh-106670
Carreau added a commit to Carreau/cpython that referenced this issue Aug 10, 2023
This lets Pdb receive an exception, instead of a traceback, and when
this is the case and the exception are chained, `exceptions` allow to
list and move between the chained exceptions.

That is to say if you have something like

    def out():
        try:
            middle()                                # B
        except Exception as e:
            raise ValueError("foo(): bar failed")   # A

    def middle():
        try:
            return inner(0)                         # D
        except Exception as e:
            raise ValueError("Middle fail") from e  # C

    def inner(x):
        1 / x                                       # E

Only A is reachable after calling `out()` and doing post mortem debug.
With this all A-E points are reachable with a combination of up/down,
and ``exception <number>``.

Note that while above each exception have a unique
`__cause__`/`__context__`, when both are present and different, each one
is separately listed (see later commits)

This also change the default behavior of ``pdb.pm()``, to receive
`sys.last_exc` so that chained exception navigation is enabled.

Closes pythongh-106670
Carreau added a commit to Carreau/cpython that referenced this issue Aug 10, 2023
This lets Pdb receive an exception, instead of a traceback, and when
this is the case and the exception are chained, `exceptions` allow to
list and move between the chained exceptions.

That is to say if you have something like

    def out():
        try:
            middle()                                # B
        except Exception as e:
            raise ValueError("foo(): bar failed")   # A

    def middle():
        try:
            return inner(0)                         # D
        except Exception as e:
            raise ValueError("Middle fail") from e  # C

    def inner(x):
        1 / x                                       # E

Only A is reachable after calling `out()` and doing post mortem debug.
With this all A-E points are reachable with a combination of up/down,
and ``exception <number>``.

This also change the default behavior of ``pdb.pm()``, to receive
`sys.last_exc` so that chained exception navigation is enabled.

Closes pythongh-106670
Carreau added a commit to Carreau/cpython that referenced this issue Aug 16, 2023
This lets Pdb receive an exception, instead of a traceback, and when
this is the case and the exception are chained, the new `exceptions` command
allows to both list (no arguments) and move between the chained exceptions.

That is to say if you have something like

    def out():
        try:
            middle()                                # B
        except Exception as e:
            raise ValueError("foo(): bar failed")   # A

    def middle():
        try:
            return inner(0)                         # D
        except Exception as e:
            raise ValueError("Middle fail") from e  # C

    def inner(x):
        1 / x                                       # E

Only A was reachable after calling `out()` and doing post mortem debug.

With this all A-E points are reachable with a combination of up/down,
and ``exception <number>``.

This also change the default behavior of ``pdb.pm()``, as well as
`python -m pdb <script.py>` to receive `sys.last_exc` so that chained
exception navigation is enabled.

We do follow the logic of the ``traceback`` module and handle the
``_context__`` and ``__cause__`` in the same way. That is to say, we try
``__cause__`` first, and if not present chain with ``__context__``. In
the same vein, if we encounter an exception that has
``__suppress_context__`` (like when ``raise ... from None``), we do stop
walking the chain.

Some implementation notes:

 - We do handle cycle in exceptions
 - cleanup of references to tracebacks are not cleared in ``forget()``, as
   ``setup()`` and ``forget()`` are both for setting a single
   exception.
 - We do not handle sub-exceptions of exception groups.

Closes pythongh-106670
AlexWaygood added a commit to Carreau/cpython that referenced this issue Aug 23, 2023
@iritkatriel iritkatriel added the stdlib Python modules in the Lib dir label Aug 24, 2023
Carreau added a commit to Carreau/cpython that referenced this issue Sep 4, 2023
The introduction of chained exception in pythongh-106676 would lead to

    File .../Lib/pdb.py", line 298, in setup
      self.curframe = self.stack[self.curindex][0]
                    ~~~~~~~~~~^^^^^^^^^^^^^^^
    IndexError: list index out of range

This fixes that by filtering exceptions that that do not have a stack.
Update tests to not use stack-less exceptions when testing another
feature, and add an explicit test on how we handle stackless exceptions.
Carreau added a commit to Carreau/cpython that referenced this issue Sep 4, 2023
The introduction of chained exception in pythongh-106676 would lead to

    File .../Lib/pdb.py", line 298, in setup
      self.curframe = self.stack[self.curindex][0]
                    ~~~~~~~~~~^^^^^^^^^^^^^^^
    IndexError: list index out of range

This fixes that by filtering exceptions that that do not have a stack.
Update tests to not use stack-less exceptions when testing another
feature, and add an explicit test on how we handle stackless exceptions.
Carreau added a commit to Carreau/cpython that referenced this issue Sep 4, 2023
The introduction of chained exception in pythongh-106676 would sometime lead to

    File .../Lib/pdb.py", line 298, in setup
      self.curframe = self.stack[self.curindex][0]
                    ~~~~~~~~~~^^^^^^^^^^^^^^^
    IndexError: list index out of range

This fixes that by filtering exceptions that that do not have a
stack/traceback. Update tests to not use stack-less exceptions when
testing another feature, and add an explicit test on how we handle
stackless exceptions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stdlib Python modules in the Lib dir type-feature A feature request or enhancement
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants