Repository navigation
Handling exceptions within a @contextmanager function doesn't clear sys.exception() #111375
Description
Bug report
Bug description:
The issue
I would expect that handling exceptions within a contextlib.contextmanager-created function would work in the same way as other functions and clear the sys.exception() after an error is handled.
import contextlib
import sys
def p(msg):
print(msg, repr(sys.exception()), sep=": ")
def ctx_gen():
p("before yield")
try:
yield
except:
p("during handling")
p("after handling")
ctx = contextlib.contextmanager(ctx_gen)
with ctx():
1/0The above example prints:
before yield: None
during handling: ZeroDivisionError('division by zero')
after handling: ZeroDivisionError('division by zero')
Whereas since the error was handled by the except: block, my expectation was:
before yield: None
during handling: ZeroDivisionError('division by zero')
after handling: None
Just working as designed?
From doing some digging, it seems like this is happening because the exception is still being handled by the _GeneratorContextManager.__exit__ function (added by the @contextlib.contextmanager decorator) that's driving the ctx_gen generator, even after the ctg_gen has handled it.
The following is a very rough approximation of how @contextlib.contextmanager drives ctx_gen:
c = ctx_gen()
next(c) # __enter__()
try:
# code inside the with block
1/0
except Exception as e: # __exit__(typ, exc, tb) for e
# throw exception into generator and expect to run to end
try:
c.throw(e)
except StopIteration:
pass
else: # __exit__(None, None, None)
# expect to run to end
try:
next(e)
except StopIteration:
passRunning the above (including the definitions from the first code block) also prints:
before yield: None
during handling: ZeroDivisionError('division by zero')
after handling: ZeroDivisionError('division by zero')
In the above code, it's more clear that the exception is still being handled by the except Exception as e: block until c.throw() returns/raises, which only happens after the generator exits. Therefore the exception is still being handled the entire time ctx_gen is running all the code after the first yield.
The fix?
Even though this behavior looks to be technically correct, it still seems unexpected and a bit of an abstraction leak.
Is this something that can be/should be fixed? Or should the behavior just be documented?
CPython versions tested on:
3.8, 3.9, 3.10, 3.11, CPython main branch
Operating systems tested on:
macOS
Activity
ericsnowcurrently commentedon Oct 31, 2023
CC @ncoghlan
iritkatriel commentedon Nov 1, 2023
This is a nice bug - everything is working as designed but the result is indeed unexpected.
I don't think it can be fixed in contextlib without changes to the interpreter, and if I'm right it's unlikely that this will use case will justify such changes.
Either way, for now at least we should mention this as a quirk of contextlib. @pR0Ps would you like to propose a doc patch?
pR0Ps commentedon Nov 2, 2023
I was messing around with this and after I wired up a C API call to
PyErr_SetHandledException(Py_None)just before theself.gen.throwline in the following code, I get the "expected" result.Before:
cpython/Lib/contextlib.py
Lines 161 to 163 in 230e8e9
After:
This essentially amounts to running Python 2's
sys.exc_clear()just before throwing the exception into the generator. While the generator still has to handle the exception that was thrown into it, it's doing it with no current exception context. This means that once it handles that thrown exception, there is no longer any exception being handled, makingsys.exception()returnNonewhich matches what a user would expect.With this change, the result of running the test case from above matches my expected result:
At the moment, my implementation is just a proof of concept that I quickly hacked together, but I'd be willing to clean it up, fix the edge cases, write tests, etc. if this is something that would be considered acceptable to add in some fashion and not too hacky.
iritkatriel commentedon Nov 2, 2023
Ah yes, that might work. You'd need to restore the exception after throw() returns. Make sure you have a test for when this whole thing is nested in another except block (and check the value of sys.exception() after the nested one).
gh-111375: Use `NULL` rather than `None` in the exception stack to in…
pythongh-111375: Use `NULL` rather than `None` in the exception stack…
pythongh-111375: Use `NULL` rather than `None` in the exception stack…
pythongh-111375: Use `NULL` rather than `None` in the exception stack…
pythongh-111375: Use `NULL` rather than `None` in the exception stack…
kumaraditya303 commentedon May 25, 2026
There's a similar issue with async context managers as well, here's a similar reproducer based on the original: