From 4b67787a644580dde252b31d11478fb5047d2b79 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 22 Apr 2026 17:04:00 +0200 Subject: [PATCH 01/12] Work on yield expression MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Bartosz Sławecki Co-authored-by: Blaise Pabon --- Doc/builtins/stdtypes.rst | 94 ++++++-- Doc/glossary.rst | 6 +- Doc/reference/expressions.rst | 410 ++++++++++++++++++++++++++-------- 3 files changed, 400 insertions(+), 110 deletions(-) diff --git a/Doc/builtins/stdtypes.rst b/Doc/builtins/stdtypes.rst index cf05f947c5908d..1ce4abdefb0680 100644 --- a/Doc/builtins/stdtypes.rst +++ b/Doc/builtins/stdtypes.rst @@ -976,32 +976,82 @@ A function that contains one or more :ref:`yield expressions ` is a :term:`generator function`. For example:: - >>> def count_to_three(): - ... yield 0 - ... yield 1 - ... yield 2 - ... yield 3 + >>> def generator_function(): + ... print("generator starting") + ... yield 123 + ... print("generator at line 4") + ... yield 456 + ... print("generator ending") Generator functions behave as regular :ref:`user-defined functions ` (for example, they have the same attributes), except that calling a generator -function returns a :ref:`generator iterator `:: - - >>> count_to_three() - +function returns a :term:`generator iterator`:: + + >>> generator_iterator = generator_function() + >>> generator_iterator + + +The iterator then allows control over the execution of the underlying function. +Initially, the function is *suspended*. +Execution starts when a :dfn:`consumer` asks the iterator to produce the next +value, for example using the :func:`next` function (which calls +the generator's :meth:`!generator.__next__()` method). +The execution starts at the beginning of the function body +and proceeds to the first :keyword:`!yield` expression. +The expression after the :keyword:`!yield` keyword is evaluated and returned +to the consumer (here, the caller of ``next``):: + + >>> first_value = next(generator_iterator) + generator starting + >>> print(first_value) + 123 + +At this point, execution of the generator function is *suspended* again. +By *suspended*, we mean that all local execution state is retained, including the +current bindings of local variables, the "instruction pointer" (index of +the :keyword:`!yield` expression at which execution is suspended), +the internal evaluation stack, and the state of any exception handling. + +When the iterator is asked to produce another value, execution is *resumed*: +the function proceeds from the :keyword:`!yield` expression where it was +suspended until the next :keyword:`!yield` expression is encountered:: + + >>> second_value = next(generator_iterator) + generator at line 4 + >>> print(second_value) + 456 + +If the generator function returns before a :keyword:`!yield` expression is +encountered (either with a :keyword:`return` statement, or implicitly at the +end of the function), the generator iterator becomes :term:`exhausted`. +All further attempts to get the next value from it will raise +a :exc:`StopIteration` exception:: + + >>> third_value = next(generator_iterator) + Traceback (most recent call last): + ... + StopIteration + >>> third_value = next(generator_iterator) + Traceback (most recent call last): + ... + StopIteration -Iterating a generator iterator executes code of the underlying -generator function, producing each :keyword:`yield`\ed value in turn:: +Commonly, an iterator is consumed via a :keyword:`for` loop, which implicitly +calls :func:`next` at the start of each iteration, and handles +:exc:`StopIteration`:: - >>> for number in count_to_three(): - ... print(number) - 0 - 1 - 2 - 3 + >>> for value in generator_function(): + ... print("value from generator:", value) + generator starting + value from generator: 123 + generator at line 4 + value from generator: 456 + generator ending - >>> list(count_to_three()) - [0, 1, 2, 3] +Note that instead of reusing the exhausted ``generator_iterator``, +this example calls ``generator_function()`` again to create a new iterator +with its own local execution state. One common use for generator functions is implementing the :meth:`~object.__iter__` method of custom iterable objects. @@ -1015,6 +1065,12 @@ For example:: >>> list(CardDeck()) ['three of clubs', 'ace of hearts'] +.. seealso:: + + :pep:`255` - Simple Generators + The proposal for adding generators and the :keyword:`yield` statement to Python. + + .. index:: pair: object; generator .. _generator-methods: diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 68bc0560911161..2accabf4195f81 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -649,7 +649,7 @@ Glossary collector that is able to detect and break reference cycles. The garbage collector can be controlled using the :mod:`gc` module. - .. index:: single: generator + .. index:: single: generator generator Informally used to mean either a :term:`generator function` or a @@ -657,7 +657,7 @@ Glossary :term:`generator function` and :term:`generator iterator` are uncommon in practice; "generator" alone is almost always sufficient. - .. index:: single: generator function + .. index:: single: generator function generator function A function which returns a :term:`generator` object. It looks like a @@ -680,7 +680,7 @@ Glossary :meth:`~generator.throw` method to raise an exception at the point where the generator was paused. See :ref:`generator-methods`. - .. index:: single: generator expression + .. index:: single: generator expression generator expression An :term:`expression` that returns an :term:`iterator`. It looks like a normal expression diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 2e0b6621eb11c8..e4f02b6d682b24 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -974,7 +974,7 @@ which yields the same values as the corresponding list comprehension:: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81] Thus, the example above is roughly equivalent to defining and calling -the following generator function:: +the following :term:`generator function`:: def make_generator_of_squares(iterator): for x in iterator: @@ -1043,8 +1043,9 @@ the implicitly nested scope. If a generator expression contains either :keyword:`!async for` clauses or :keyword:`await` expressions it is called an :dfn:`asynchronous generator expression`. -An asynchronous generator expression returns a new asynchronous generator -object, which is an asynchronous iterator (see :ref:`async-iterators`). +An asynchronous generator expression returns a new +:term:`asynchronous generator iterator`, which is an +:ref:`asynchronous iterator `. The formal grammar for generator expressions is: @@ -1065,116 +1066,322 @@ The formal grammar for generator expressions is: ``yield`` and ``yield from`` prohibited in the implicitly nested scope. -.. _yieldexpr: - -Yield expressions ------------------ - .. index:: pair: keyword; yield pair: keyword; from pair: yield; expression pair: generator; function -.. productionlist:: python-grammar - yield_atom: "(" `yield_expression` ")" - yield_from: "yield" "from" `expression` - yield_expression: "yield" `yield_list` | `yield_from` +.. _yieldexpr: -The yield expression is used when defining a :term:`generator` function -or an :term:`asynchronous generator` function and -thus can only be used in the body of a function definition. Using a yield -expression in a function's body causes that function to be a generator function, -and using it in an :keyword:`async def` function's body causes that -coroutine function to be an asynchronous generator function. For example:: +Yield expressions +----------------- - def gen(): # defines a generator function - yield 123 +The yield expression may only occur syntactically nested in a function +definition, not within a nested class definition. +See :ref:`yield-expression-placement` for details. - async def agen(): # defines an asynchronous generator function - yield 123 +A function that contains one or more :ref:`yield expressions ` +is a :term:`generator function`. +At runtime, calling the function returns a :term:`generator iterator`, +which produces values after the :keyword:`yield` keyword in turn. +For example:: -Due to their side effects on the containing scope, ``yield`` expressions -are not permitted as part of the implicitly defined scopes used to -implement comprehensions and generator expressions. + >>> def count_to_three(): + ... yield 0 + ... yield 1 + ... yield 2 + ... yield 3 -.. versionchanged:: 3.8 - Yield expressions prohibited in the implicitly nested scopes used to - implement comprehensions and generator expressions. + >>> for number in count_to_three(): + ... print(number) + 0 + 1 + 2 + 3 -Generator functions are described below, while asynchronous generator -functions are described separately in section -:ref:`asynchronous-generator-functions`. - -When a generator function is called, it returns an iterator known as a -generator. That generator then controls the execution of the generator -function. The execution starts when one of the generator's methods is called. -At that time, the execution proceeds to the first yield expression, where it is -suspended again, returning the value of :token:`~python-grammar:yield_list` -to the generator's caller, -or ``None`` if :token:`~python-grammar:yield_list` is omitted. -By suspended, we mean that all local state is -retained, including the current bindings of local variables, the instruction -pointer, the internal evaluation stack, and the state of any exception handling. -When the execution is resumed by calling one of the generator's methods, the -function can proceed exactly as if the yield expression were just another -external call. The value of the yield expression after resuming depends on the -method which resumed the execution. If :meth:`~generator.__next__` is used -(typically via either a :keyword:`for` or the :func:`next` builtin) then the -result is :const:`None`. Otherwise, if :meth:`~generator.send` is used, then -the result will be the value passed in to that method. - -.. index:: single: coroutine - -All of this makes generator functions quite similar to coroutines; they yield -multiple times, they have more than one entry point and their execution can be -suspended. The only difference is that a generator function cannot control -where the execution should continue after it yields; the control is always -transferred to the generator's caller. - -Yield expressions are allowed anywhere in a :keyword:`try` construct. If the -generator is not resumed before it is -finalized (by reaching a zero reference count or by being garbage collected), -the generator-iterator's :meth:`~generator.close` method will be called, -allowing any pending :keyword:`finally` clauses to execute. +See :ref:`generator-types` for details on runtime behavior, and other +ways to control control flow through the function. -.. index:: - single: from; yield from expression +If the enclosing function is defined using ``async def``, it becomes an +:term:`asynchronous generator function` instead. +For example:: -When ``yield from `` is used, the supplied expression must be an -iterable. The values produced by iterating that iterable are passed directly -to the caller of the current generator's methods. Any values passed in with -:meth:`~generator.send` and any exceptions passed in with -:meth:`~generator.throw` are passed to the underlying iterator if it has the -appropriate methods. If this is not the case, then :meth:`~generator.send` -will raise :exc:`AttributeError` or :exc:`TypeError`, while -:meth:`~generator.throw` will just raise the passed in exception immediately. + async def agen(): # defines an asynchronous generator function + yield 123 -When the underlying iterator is complete, the :attr:`~StopIteration.value` -attribute of the raised :exc:`StopIteration` instance becomes the value of -the yield expression. It can be either set explicitly when raising -:exc:`StopIteration`, or automatically when the subiterator is a generator -(by returning a value from the subgenerator). +Asynchronous generator functions are described separately +:ref:`below `. -.. versionchanged:: 3.3 - Added ``yield from `` to delegate control flow to a subiterator. -The parentheses may be omitted when the yield expression is the sole expression -on the right hand side of an assignment statement. +:keyword:`yield` without a value +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. seealso:: +If the value after a :keyword:`yield` keyword is omitted, the generator +iterator produces ``None``:: + + >>> def generator_function(): + ... yield 123 + ... yield - :pep:`255` - Simple Generators - The proposal for adding generators and the :keyword:`yield` statement to Python. + >>> for value in generator_function(): + ... print("value from generator:", value) + value from generator: 123 + value from generator: None + + +Value of the :keyword:`yield` expression +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +``yield`` can be used as an expression, for example as the value of +an assignment:: + + >>> def generator_function(): + ... value = yield 123 + ... print("value of yield is", value) + +At runtime, when using the basic iteration protocol (:keyword:`for` or +:func:`next`), the value of a :keyword:`yield` expression is ``None``:: + + >>> for value in generator_function(): + ... pass + value of yield is: None + +A different value can be given by using the generator iterator's +:meth:`~generator.send` method instead of :keyword:`for` or :func:`!next`:: + + >>> generator_iterator = generator_function() + >>> next(generator_iterator) + 123 + >>> generator_iterator.send("different") + value of yield is: different + +Note that :meth:`~generator.send` with a non-``None`` argument can only +be used when the generator is suspended at a :keyword:`yield` expression, +not with a just-started iterator that is suspended at the beginning of the +generator function:: + + >>> generator_iterator = generator_function() + >>> generator_iterator.send("different") + Traceback (most recent call last): + ... + generator_iterator.send("different") + ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^ + TypeError: can't send non-None value to a just-started generator + +.. seealso:: :pep:`342` - Coroutines via Enhanced Generators The proposal to enhance the API and syntax of generators, making them usable as simple coroutines. + +Parentheses around :keyword:`yield` expressions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Yield expressions must be enclosed in parentheses except in the following +common situations (which are used in the examples above): + +- the :ref:`yield statement `'s only expression, or +- the only expression on the right-hand side of an + :ref:`assignment statement `. + +For example:: + + def gen(): + yield 123 # yield statement: parentheses not required + + print("first value:", (yield 456)) # parentheses are required + + value = yield 789 # assignment statement: parentheses not required + print("second value:", value) + + value = (yield 147) + 258 # part of larger expression: parens required + print("third value:", value) + + +:keyword:`yield` expressions and :keyword:`try` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Yield expressions are allowed anywhere in a :keyword:`try` construct. + +If the generator iterator is not resumed before it is destroyed +(garbage collected), its :meth:`~generator.close` method will be called, +raising a :exc:`GeneratorExit` at the :keyword:`!yield` point where the +function is suspended. +This allows any pending :keyword:`finally` clauses to execute. + +.. impl-detail:: + + In the following example, ``iterator`` is garbage collected immediately + after the ``del iterator`` statement, triggering the ``finally`` block:: + + >>> def gen(): + ... try: + ... yield 123 + ... finally: + ... print('shutting down') + ... + >>> iterator = gen() + >>> next(iterator) + 123 + >>> del iterator + shutting down + + On non-CPython implementations, or future versions of CPython, + garbage collection (and thus printing the message) may happen at any + later point, or even not at all. + +Note that yielding a value after :meth:`~generator.close` raises +:exc:`!GeneratorExit` is an error:: + + >>> def gen(): + ... try: + ... yield 123 + ... except GeneratorExit: + ... pass # ignore the exception + ... yield 456 + ... + >>> iterator = gen() + >>> next(iterator) + 123 + >>> iterator.close() + Traceback (most recent call last): + File "", line 1, in + iterator.close() + ~~~~~~~~^^ + RuntimeError: generator ignored GeneratorExit + + +.. index:: + single: from; yield from expression + +:keyword:`yield from` expressions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When :samp:`yield from {subiterable}` is used, *subiterable* must evaluate to +an iterable. +At runtime, the current generator's operations are :dfn:`delegated` to the +subiterable: the values produced by the subiterable are passed directly +to the consumer of the current generator. + +For simple iterables, :samp:`yield from {subiterable}` is essentially +a shortened form of :samp:`for item in {subiterable}: yield item`:: + + >>> hand_movements = ['put hands up', 'put hands down'] + + >>> def dance(): + ... yield from hand_movements + ... yield 'turn around' + ... yield from hand_movements + + >>> for move in dance(): + ... print(move) + put hands up + put hands down + turn around + put hands up + put hands down + +However, unlike an ordinary loop, ``yield from`` delegates the entire generator +protocol: it allows the subiterable to receive sent and thrown values directly +from the consumer, and return a final value to the delegating function. + +When the subiterable is complete, the :attr:`~StopIteration.value` +attribute of the raised :exc:`StopIteration` instance becomes the value of +the ``yield from`` expression. +The value can be either set explicitly when raising +:exc:`StopIteration`, or, when the subiterator is a generator, +by returning a value from the subgenerator's underlying function:: + + TODO: find a good example + + >>> def subgen(): + ... yield 1 + ... yield 2 + ... return 3 + + >>> def maingen(): + ... result = yield from subgen() + ... yield result + ... yield from subgen() + + >>> list(maingen()) + [1, 2, 3, 1, 2] + +Any values passed in with :meth:`~generator.send` and any exceptions passed +in with :meth:`~generator.throw` are passed to the subiterator if it +has the appropriate methods. +If this is not the case, then :meth:`~generator.send` will raise +:exc:`AttributeError` or :exc:`TypeError`, while :meth:`~generator.throw` +will raise the passed in exception immediately. + +.. versionchanged:: 3.3 + Added ``yield from `` to delegate control flow to a subiterator. + +.. seealso:: + :pep:`380` - Syntax for Delegating to a Subgenerator The proposal to introduce the :token:`~python-grammar:yield_from` syntax, making delegation to subgenerators easy. +.. _yield-expression-placement: + +Where :keyword:`yield` expressions are allowed +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The yield expression can only be used in the body of a function definition. +More precisely, it can only be used when the closest enclosing *block* +is a function definition, not a module-level or class block. +(See :ref:`prog_structure` for an explanation of blocks.) +For example:: + + class C: + # yield is not allowed here + + def foo(self): + yield 1 # this makes `foo` a generator function + + class NestedClass: + # yield is not allowed here + pass + + def nested_function(): + # this function is *not* a generator + + def double_nested(): + yield 2 # this makes `double_nested` a generator function + +Due to their side effects on the containing scope, ``yield`` expressions +are not permitted as part of the implicitly defined scopes used to +implement comprehensions and generator expressions. + +For :keyword:`yield from` expressions, the same restrictions apply. + +.. versionchanged:: 3.8 + Yield expressions prohibited in the implicitly nested scopes used to + implement comprehensions and generator expressions. + + +Formal grammar of :keyword:`yield` expressions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. grammar-snippet:: + :group: python-grammar + + yield_atom: "(" `yield_expression` ")" + + yield_expression: + | 'yield' 'from' `expression` + | 'yield' [`yield_list`] + + yield_list: + | `expression_list` + | `starred_expression` "," [`starred_expression_list`] + + +.. seealso:: + :pep:`525` - Asynchronous Generators The proposal that expanded on :pep:`492` by adding generator capabilities to coroutine functions. @@ -1211,8 +1418,36 @@ generator functions:: >>> generator.close() Don't forget to clean up when 'close()' is called. -For examples using ``yield from``, see :ref:`pep-380` in "What's New in -Python." +Here is a more complex example that demonstrates ``yield from``, including +delegation of sent values and receiving the value returned by +a subgenerator function:: + + >>> def accumulate(): + ... tally = 0 + ... while True: + ... next = yield + ... if next is None: + ... return tally + ... tally += next + ... + >>> def gather_tallies(tallies): + ... while True: + ... tally = yield from accumulate() + ... tallies.append(tally) + ... + >>> tallies = [] + >>> acc = gather_tallies(tallies) + >>> next(acc) # Ensure the accumulator is ready to accept values + >>> for i in range(4): + ... acc.send(i) + ... + >>> acc.send(None) # Finish the first tally + >>> for i in range(5): + ... acc.send(i) + ... + >>> acc.send(None) # Finish the second tally + >>> tallies + [6, 10] .. _asynchronous-generator-functions: @@ -2475,7 +2710,6 @@ Expression lists flexible_expression_list: `flexible_expression` ("," `flexible_expression`)* [","] starred_expression_list: `starred_expression` ("," `starred_expression`)* [","] expression_list: `expression` ("," `expression`)* [","] - yield_list: `expression_list` | `starred_expression` "," [`starred_expression_list`] .. index:: pair: object; tuple From 590b50f4b0850a06df22139a07e3fbb21f29aa25 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 16 Sep 2026 16:44:28 +0200 Subject: [PATCH 02/12] Merge "yield statements" into "expression statements" --- Doc/reference/expressions.rst | 54 +++++++++++++++++++++-------- Doc/reference/simple_stmts.rst | 63 +++++++++++----------------------- 2 files changed, 59 insertions(+), 58 deletions(-) diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index e4f02b6d682b24..f7a9a7e300d730 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -1072,19 +1072,27 @@ The formal grammar for generator expressions is: pair: yield; expression pair: generator; function +.. _yield: .. _yieldexpr: -Yield expressions ------------------ +.. index:: + single: generator; function + single: generator; iterator + single: function; generator + pair: exception; StopIteration + +:keyword:`!yield` expressions +----------------------------- -The yield expression may only occur syntactically nested in a function -definition, not within a nested class definition. +The :keyword:`!yield` expression may only occur syntactically nested in +a function definition, not within a nested class definition. See :ref:`yield-expression-placement` for details. A function that contains one or more :ref:`yield expressions ` is a :term:`generator function`. -At runtime, calling the function returns a :term:`generator iterator`, -which produces values after the :keyword:`yield` keyword in turn. +At runtime, calling the function returns a :term:`generator iterator` +which, when iterated, runs the underlying function and produces values +after the :keyword:`!yield` keywords. For example:: >>> def count_to_three(): @@ -1100,7 +1108,7 @@ For example:: 2 3 -See :ref:`generator-types` for details on runtime behavior, and other +See :ref:`generator-types` for details on runtime behavior and ways to control control flow through the function. If the enclosing function is defined using ``async def``, it becomes an @@ -1176,28 +1184,44 @@ generator function:: usable as simple coroutines. +.. _yield-parentheses: + Parentheses around :keyword:`yield` expressions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Yield expressions must be enclosed in parentheses except in the following common situations (which are used in the examples above): -- the :ref:`yield statement `'s only expression, or +- an :ref:`expression statement `'s only expression, or - the only expression on the right-hand side of an :ref:`assignment statement `. For example:: - def gen(): - yield 123 # yield statement: parentheses not required + def yields_in_expression_statements(): + + # single yield expression: parentheses NOT required + yield 123 + + # two yield expressions: parentheses required + (yield 456, yield 789) + + # part of larger expression: parentheses required + print("value:", (yield 147)) + + def yields_in_assignment_statements(): - print("first value:", (yield 456)) # parentheses are required + # single yield expression: parentheses NOT required + value = yield 258 + print("value:", value) - value = yield 789 # assignment statement: parentheses not required - print("second value:", value) + # two yield expressions: parentheses required + values = (yield 456, yield 789) + print("values:", value) - value = (yield 147) + 258 # part of larger expression: parens required - print("third value:", value) + # part of larger expression: parentheses required + value = (yield 369) + 159 + print("value:", value) :keyword:`yield` expressions and :keyword:`try` diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index a964b43ebee174..750539bb8a50a9 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -20,7 +20,6 @@ simple statements is: : | `pass_stmt` : | `del_stmt` : | `return_stmt` - : | `yield_stmt` : | `raise_stmt` : | `break_stmt` : | `continue_stmt` @@ -32,6 +31,7 @@ simple statements is: .. _exprstmts: +.. _yield statements: Expression statements ===================== @@ -44,12 +44,7 @@ Expression statements Expression statements are used (mostly interactively) to compute and write a value, or (usually) to call a procedure (a function that returns no meaningful result; in Python, procedures return the value ``None``). Other uses of -expression statements are allowed and occasionally useful. The syntax for an -expression statement is: - -.. productionlist:: python-grammar - expression_stmt: `starred_expression` - +expression statements are allowed and occasionally useful. An expression statement evaluates the expression list (which may be a single expression). @@ -67,6 +62,24 @@ using the built-in :func:`repr` function and the resulting string is written to standard output on a line by itself (except if the result is ``None``, so that procedure calls do not cause any output.) +The formal grammar for an expression statement is: + +.. index:: pair: statement; yield + +.. grammar-snippet:: + :group: python-grammar + + expression_stmt: `starred_expression` | `yield_expression` + +.. note:: + + The formal grammar includes as special case for a single :keyword:`!yield` + expression, without parentheses around it. + This is sometimes called a :dfn:`yield statement`. + For complete documentation of :keyword:`!yield` semantics, see + the :ref:`yieldexpr` section. + + .. _assignment: Assignment statements @@ -512,42 +525,6 @@ indicates that the asynchronous generator is done and will cause :exc:`StopAsyncIteration` to be raised. A non-empty :keyword:`!return` statement is a syntax error in an asynchronous generator function. -.. _yield: - -The :keyword:`!yield` statement -=============================== - -.. index:: - pair: statement; yield - single: generator; function - single: generator; iterator - single: function; generator - pair: exception; StopIteration - -.. productionlist:: python-grammar - yield_stmt: `yield_expression` - -A :keyword:`yield` statement is semantically equivalent to a :ref:`yield -expression `. The ``yield`` statement can be used to omit the -parentheses that would otherwise be required in the equivalent yield expression -statement. For example, the yield statements :: - - yield - yield from - -are equivalent to the yield expression statements :: - - (yield ) - (yield from ) - -Yield expressions and statements are only used when defining a :term:`generator` -function, and are only used in the body of the generator function. Using :keyword:`yield` -in a function definition is sufficient to cause that definition to create a -generator function instead of a normal function. - -For full details of :keyword:`yield` semantics, refer to the -:ref:`yieldexpr` section. - .. _raise: The :keyword:`!raise` statement From 7b96ea2043c54e1d04662bf22d4651868b5a4faf Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 23 Sep 2026 16:32:31 +0200 Subject: [PATCH 03/12] Reword the intro a bit --- Doc/reference/expressions.rst | 35 ++++++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index f7a9a7e300d730..2f46b65f8e677b 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -1084,15 +1084,27 @@ The formal grammar for generator expressions is: :keyword:`!yield` expressions ----------------------------- -The :keyword:`!yield` expression may only occur syntactically nested in -a function definition, not within a nested class definition. -See :ref:`yield-expression-placement` for details. - -A function that contains one or more :ref:`yield expressions ` -is a :term:`generator function`. -At runtime, calling the function returns a :term:`generator iterator` -which, when iterated, runs the underlying function and produces values -after the :keyword:`!yield` keywords. +.. This is a high-level conceptual overview. + The linked ":ref:`generator-types`" section (in the Builtins chapter) + has the details on runtime behaviour; + further sections *here* give details on the syntax. + +The :keyword:`!yield` keyword introduces a :dfn:`yield expression`. +:keyword:`!yield` expressions may only occur inside function definitions; +see :ref:`yield-expression-placement` for details. + +When a function definition contains one or more +yield expressions, it defines a :term:`generator function`. +At runtime, calling such a function does not run its code directly, but instead +returns a :term:`generator iterator` which controls execution of the underlying +function. + +In :keyword:`!yield` expressions, the :keyword:`!yield` keyword is usually +followed by an expression. +At runtime, each time the generator iterator is asked to produce the next +value, execution proceeds to the next :keyword:`!yield` keyword, its expression +is evaluated, and the result is used as the iterator's next value. + For example:: >>> def count_to_three(): @@ -1108,10 +1120,11 @@ For example:: 2 3 + See :ref:`generator-types` for details on runtime behavior and -ways to control control flow through the function. +ways to control execution of the underlying function. -If the enclosing function is defined using ``async def``, it becomes an +If the underlying function is defined using ``async def``, it becomes an :term:`asynchronous generator function` instead. For example:: From c74219d3d9cb86a6360b2335e63e65b10f212e6e Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 23 Sep 2026 16:46:10 +0200 Subject: [PATCH 04/12] Work more on the intro --- Doc/reference/expressions.rst | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 2f46b65f8e677b..7479195dac6ac2 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -1093,14 +1093,14 @@ The :keyword:`!yield` keyword introduces a :dfn:`yield expression`. :keyword:`!yield` expressions may only occur inside function definitions; see :ref:`yield-expression-placement` for details. -When a function definition contains one or more -yield expressions, it defines a :term:`generator function`. +A function that contains one or more yield expressions +is a :term:`generator function`. At runtime, calling such a function does not run its code directly, but instead returns a :term:`generator iterator` which controls execution of the underlying function. -In :keyword:`!yield` expressions, the :keyword:`!yield` keyword is usually -followed by an expression. +The :keyword:`!yield` keyword is usually followed by +an :ref:`expression `. At runtime, each time the generator iterator is asked to produce the next value, execution proceeds to the next :keyword:`!yield` keyword, its expression is evaluated, and the result is used as the iterator's next value. @@ -1108,10 +1108,11 @@ is evaluated, and the result is used as the iterator's next value. For example:: >>> def count_to_three(): + ... """A generator that produces the numbers 0 to 3""" ... yield 0 ... yield 1 ... yield 2 - ... yield 3 + ... yield 1 + 2 >>> for number in count_to_three(): ... print(number) @@ -1128,7 +1129,8 @@ If the underlying function is defined using ``async def``, it becomes an :term:`asynchronous generator function` instead. For example:: - async def agen(): # defines an asynchronous generator function + async def agen(): + """An asynchronous generator that produces a single value""" yield 123 Asynchronous generator functions are described separately From cb4bfc924890bc435c0fba3b32587e82fa8c07d0 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 23 Sep 2026 17:29:54 +0200 Subject: [PATCH 05/12] WIP --- Doc/builtins/stdtypes.rst | 6 ++++ Doc/reference/expressions.rst | 67 +++++++++++++++++------------------ 2 files changed, 38 insertions(+), 35 deletions(-) diff --git a/Doc/builtins/stdtypes.rst b/Doc/builtins/stdtypes.rst index 1ce4abdefb0680..c0894d359ab66a 100644 --- a/Doc/builtins/stdtypes.rst +++ b/Doc/builtins/stdtypes.rst @@ -1181,6 +1181,12 @@ Calling any of the generator methods (:meth:`~generator.__next__`, while one of these methods is already executing raises a :exc:`ValueError` exception. +.. seealso:: + + :pep:`342` - Coroutines via Enhanced Generators + The proposal to add :meth:`!send`, :meth:`!throw` and :meth:`!close`, + making generators usable as simple coroutines. + .. index:: pair: object; asynchronous-generator .. _asynchronous-generator-methods: diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 7479195dac6ac2..78b93c5a38b9ff 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -1115,12 +1115,11 @@ For example:: ... yield 1 + 2 >>> for number in count_to_three(): - ... print(number) - 0 - 1 - 2 - 3 - + ... print("got:", number) + got: 0 + got: 1 + got: 2 + got: 3 See :ref:`generator-types` for details on runtime behavior and ways to control execution of the underlying function. @@ -1140,24 +1139,24 @@ Asynchronous generator functions are described separately :keyword:`yield` without a value ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -If the value after a :keyword:`yield` keyword is omitted, the generator +If the expression after a :keyword:`yield` keyword is omitted, the generator iterator produces ``None``:: - >>> def generator_function(): + >>> def generator(): ... yield 123 ... yield - >>> for value in generator_function(): - ... print("value from generator:", value) - value from generator: 123 - value from generator: None + >>> for value in generator(): + ... print("got:", value) + got: 123 + got: None -Value of the :keyword:`yield` expression -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Value of the :keyword:`!yield` expression +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -``yield`` can be used as an expression, for example as the value of -an assignment:: +Like other expressions, :keyword:`!yield` expressions +can be used in assignments:: >>> def generator_function(): ... value = yield 123 @@ -1171,32 +1170,23 @@ At runtime, when using the basic iteration protocol (:keyword:`for` or value of yield is: None A different value can be given by using the generator iterator's -:meth:`~generator.send` method instead of :keyword:`for` or :func:`!next`:: +:meth:`~generator.send` method instead of :keyword:`!for` or :func:`!next`:: >>> generator_iterator = generator_function() >>> next(generator_iterator) 123 + >>> # The generator is now suspended at the "yield 123" expression. + >>> # Provide a value for it: >>> generator_iterator.send("different") value of yield is: different -Note that :meth:`~generator.send` with a non-``None`` argument can only -be used when the generator is suspended at a :keyword:`yield` expression, -not with a just-started iterator that is suspended at the beginning of the -generator function:: - - >>> generator_iterator = generator_function() - >>> generator_iterator.send("different") - Traceback (most recent call last): - ... - generator_iterator.send("different") - ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^ - TypeError: can't send non-None value to a just-started generator - .. seealso:: - :pep:`342` - Coroutines via Enhanced Generators - The proposal to enhance the API and syntax of generators, making them - usable as simple coroutines. + This feature was originally added in :pep:`342` as a way to implement + coroutines. + While it still works, :term:`coroutine functions ` + (using :keyword:`async` and :keyword:`await`) offer a better way to + do this. .. _yield-parentheses: @@ -1204,8 +1194,15 @@ generator function:: Parentheses around :keyword:`yield` expressions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Yield expressions must be enclosed in parentheses except in the following -common situations (which are used in the examples above): +When a :keyword:`!yield` expressions appears as part of a larger expression, +it must be enclosed in parentheses. +For example:: + + def adding_generator(): + (yield 123) + (yield 456) + +More precisely, parentheses can be only omitted when the :keyword:`yield` +expression is: - an :ref:`expression statement `'s only expression, or - the only expression on the right-hand side of an From 971a2bc04b14c8501e20e7f6f1ecf65f344eb2ee Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 23 Sep 2026 17:46:00 +0200 Subject: [PATCH 06/12] "Using yield as an expression" Co-authored-by: Blaise Pabon --- Doc/reference/expressions.rst | 78 ++++++++++++++--------------------- 1 file changed, 32 insertions(+), 46 deletions(-) diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 78b93c5a38b9ff..8914550a92a0a9 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -1152,8 +1152,21 @@ iterator produces ``None``:: got: None -Value of the :keyword:`!yield` expression -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Using :keyword:`!yield` as an expression +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Usually, :keyword:`!yield` expressions are used alone on a line, +as in the examples above. +However, like other expressions, they can also be used in assignments and +more complex expressions. + +.. seealso:: + + This feature was originally added in :pep:`342` as a way to implement + coroutines. + While it still works, :term:`coroutine functions ` + (using :keyword:`async` and :keyword:`await`) offer a better way to + do this. Like other expressions, :keyword:`!yield` expressions can be used in assignments:: @@ -1180,26 +1193,26 @@ A different value can be given by using the generator iterator's >>> generator_iterator.send("different") value of yield is: different -.. seealso:: - - This feature was originally added in :pep:`342` as a way to implement - coroutines. - While it still works, :term:`coroutine functions ` - (using :keyword:`async` and :keyword:`await`) offer a better way to - do this. - - -.. _yield-parentheses: - -Parentheses around :keyword:`yield` expressions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + Traceback (most recent call last): + StopIteration -When a :keyword:`!yield` expressions appears as part of a larger expression, -it must be enclosed in parentheses. +A :keyword:`!yield` expression can also appear as part of a larger expression. +In this case, it must be enclosed in parentheses. For example:: - def adding_generator(): - (yield 123) + (yield 456) + >>> def adding_generator(): + ... sum_of_yields = (yield "first number") + (yield "second number") + ... print("The sum is:", sum_of_yields) + >>> iterator = adding_generator() + >>> next(iterator) + "first number" + >>> iterator.send(1) + "second number" + >>> iterator.send(2) + The sum is: 3 + + Traceback (most recent call last): + StopIteration More precisely, parentheses can be only omitted when the :keyword:`yield` expression is: @@ -1208,33 +1221,6 @@ expression is: - the only expression on the right-hand side of an :ref:`assignment statement `. -For example:: - - def yields_in_expression_statements(): - - # single yield expression: parentheses NOT required - yield 123 - - # two yield expressions: parentheses required - (yield 456, yield 789) - - # part of larger expression: parentheses required - print("value:", (yield 147)) - - def yields_in_assignment_statements(): - - # single yield expression: parentheses NOT required - value = yield 258 - print("value:", value) - - # two yield expressions: parentheses required - values = (yield 456, yield 789) - print("values:", value) - - # part of larger expression: parentheses required - value = (yield 369) + 159 - print("value:", value) - :keyword:`yield` expressions and :keyword:`try` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From 0f6ae3108a8973f5c0166374c34cd076cfdb7882 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 23 Sep 2026 17:56:33 +0200 Subject: [PATCH 07/12] Move "yield expressions in try" to stdtypes --- Doc/builtins/stdtypes.rst | 51 +++++++++++++++++++++++++++++++++ Doc/reference/expressions.rst | 53 ----------------------------------- 2 files changed, 51 insertions(+), 53 deletions(-) diff --git a/Doc/builtins/stdtypes.rst b/Doc/builtins/stdtypes.rst index c0894d359ab66a..ed9b9bfffcd995 100644 --- a/Doc/builtins/stdtypes.rst +++ b/Doc/builtins/stdtypes.rst @@ -1070,6 +1070,57 @@ For example:: :pep:`255` - Simple Generators The proposal for adding generators and the :keyword:`yield` statement to Python. +Generator finalization +^^^^^^^^^^^^^^^^^^^^^^ + +If a generator iterator is not resumed before it is destroyed +(garbage collected), its :meth:`~generator.close` method will be called, +raising a :exc:`GeneratorExit` at the :keyword:`!yield` point where the +function is suspended. +If the :keyword:`!yield` point is in a :keyword:`try` construct, +pending :keyword:`finally` clauses execute. (:keyword:`except GeneratorExit` and + +.. admonition:: CPython implementation detail + + In the following example, ``iterator`` is garbage collected immediately + after the ``del iterator`` statement, triggering the ``finally`` block:: + + >>> def gen(): + ... try: + ... yield 123 + ... finally: + ... print('shutting down') + ... + >>> iterator = gen() + >>> next(iterator) + 123 + >>> del iterator + shutting down + + On non-CPython implementations, or future versions of CPython, + garbage collection (and thus printing the message) may happen at any + later point, or even not at all. + +Note that yielding a value after :meth:`~generator.close` raises +:exc:`!GeneratorExit` is an error:: + + >>> def gen(): + ... try: + ... yield 123 + ... except GeneratorExit: + ... pass # ignore the exception + ... yield 456 + ... + >>> iterator = gen() + >>> next(iterator) + 123 + >>> iterator.close() + Traceback (most recent call last): + File "", line 1, in + iterator.close() + ~~~~~~~~^^ + RuntimeError: generator ignored GeneratorExit + .. index:: pair: object; generator diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 8914550a92a0a9..937eb5df06a8f0 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -1222,59 +1222,6 @@ expression is: :ref:`assignment statement `. -:keyword:`yield` expressions and :keyword:`try` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Yield expressions are allowed anywhere in a :keyword:`try` construct. - -If the generator iterator is not resumed before it is destroyed -(garbage collected), its :meth:`~generator.close` method will be called, -raising a :exc:`GeneratorExit` at the :keyword:`!yield` point where the -function is suspended. -This allows any pending :keyword:`finally` clauses to execute. - -.. impl-detail:: - - In the following example, ``iterator`` is garbage collected immediately - after the ``del iterator`` statement, triggering the ``finally`` block:: - - >>> def gen(): - ... try: - ... yield 123 - ... finally: - ... print('shutting down') - ... - >>> iterator = gen() - >>> next(iterator) - 123 - >>> del iterator - shutting down - - On non-CPython implementations, or future versions of CPython, - garbage collection (and thus printing the message) may happen at any - later point, or even not at all. - -Note that yielding a value after :meth:`~generator.close` raises -:exc:`!GeneratorExit` is an error:: - - >>> def gen(): - ... try: - ... yield 123 - ... except GeneratorExit: - ... pass # ignore the exception - ... yield 456 - ... - >>> iterator = gen() - >>> next(iterator) - 123 - >>> iterator.close() - Traceback (most recent call last): - File "", line 1, in - iterator.close() - ~~~~~~~~^^ - RuntimeError: generator ignored GeneratorExit - - .. index:: single: from; yield from expression From 68fa79461ef087be4c3f8af007506a734f905181 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 7 Oct 2026 16:46:50 +0200 Subject: [PATCH 08/12] Go over "yield from expressions" Co-authored-by: Blaise Pabon --- Doc/reference/expressions.rst | 59 ++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 26 deletions(-) diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 937eb5df06a8f0..c48bf36a6c08f0 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -1196,6 +1196,8 @@ A different value can be given by using the generator iterator's Traceback (most recent call last): StopIteration +.. _yield-expression-parentheses: + A :keyword:`!yield` expression can also appear as part of a larger expression. In this case, it must be enclosed in parentheses. For example:: @@ -1225,17 +1227,24 @@ expression is: .. index:: single: from; yield from expression -:keyword:`yield from` expressions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +:keyword:`!yield from` expressions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +An expression of the form :samp:`yield from {subiterable}` is +a :keyword:`!yield from` expression. + +Like a :keyword:`yield` expression, a :keyword:`!yield from` expression +is only allowed in functions and turns the enclosing function into a generator. +It must be parenthesized in some contexts; :ref:`the rules ` +are the same as for :keyword:`!yield` expressions. -When :samp:`yield from {subiterable}` is used, *subiterable* must evaluate to -an iterable. -At runtime, the current generator's operations are :dfn:`delegated` to the -subiterable: the values produced by the subiterable are passed directly -to the consumer of the current generator. +At runtime, *subiterable* -- the expression after the :keyword:`!yield from` +keywords -- must evaluate to an iterable. For simple iterables, :samp:`yield from {subiterable}` is essentially -a shortened form of :samp:`for item in {subiterable}: yield item`:: +a shortened form of :samp:`for item in {subiterable}: yield item`. +Values produced by *subiterable* are passed directly +to the consumer of the current generator. For example:: >>> hand_movements = ['put hands up', 'put hands down'] @@ -1252,18 +1261,24 @@ a shortened form of :samp:`for item in {subiterable}: yield item`:: put hands up put hands down -However, unlike an ordinary loop, ``yield from`` delegates the entire generator -protocol: it allows the subiterable to receive sent and thrown values directly -from the consumer, and return a final value to the delegating function. +However, unlike an ordinary loop, :keyword:`!yield from` delegates the +entire generator protocol. +Any values passed in with :meth:`~generator.send` and any exceptions passed +in with :meth:`~generator.throw` and :func:`~generator.close` are passed +to the subiterator if it has the appropriate methods. +If this is not the case, then :meth:`~generator.send` will raise +:exc:`AttributeError` or :exc:`TypeError`, while :meth:`~generator.throw` +and :func:`~generator.close` will raise the passed in exception immediately. -When the subiterable is complete, the :attr:`~StopIteration.value` +When a :keyword:`!yield from` expression's *subiterable* is complete, +the :attr:`~StopIteration.value` attribute of the raised :exc:`StopIteration` instance becomes the value of -the ``yield from`` expression. +the :keyword:`!yield from` expression. The value can be either set explicitly when raising :exc:`StopIteration`, or, when the subiterator is a generator, -by returning a value from the subgenerator's underlying function:: +by returning a value from the subgenerator's underlying function: - TODO: find a good example +.. code-block:: pycon >>> def subgen(): ... yield 1 @@ -1272,18 +1287,10 @@ by returning a value from the subgenerator's underlying function:: >>> def maingen(): ... result = yield from subgen() - ... yield result - ... yield from subgen() + ... yield ('result:', result) >>> list(maingen()) - [1, 2, 3, 1, 2] - -Any values passed in with :meth:`~generator.send` and any exceptions passed -in with :meth:`~generator.throw` are passed to the subiterator if it -has the appropriate methods. -If this is not the case, then :meth:`~generator.send` will raise -:exc:`AttributeError` or :exc:`TypeError`, while :meth:`~generator.throw` -will raise the passed in exception immediately. + [1, 2, ('result:', 3)] .. versionchanged:: 3.3 Added ``yield from `` to delegate control flow to a subiterator. @@ -1325,7 +1332,7 @@ Due to their side effects on the containing scope, ``yield`` expressions are not permitted as part of the implicitly defined scopes used to implement comprehensions and generator expressions. -For :keyword:`yield from` expressions, the same restrictions apply. +For :keyword:`!yield from` expressions, the same restrictions apply. .. versionchanged:: 3.8 Yield expressions prohibited in the implicitly nested scopes used to From 827852dc98673f65597c77501121f05f27dbd06f Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 7 Oct 2026 16:54:23 +0200 Subject: [PATCH 09/12] The last two sections before acyncgen --- Doc/reference/expressions.rst | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index c48bf36a6c08f0..e121040ce7fb9e 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -1306,7 +1306,8 @@ by returning a value from the subgenerator's underlying function: Where :keyword:`yield` expressions are allowed ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The yield expression can only be used in the body of a function definition. +A :keyword:`yield` expression can only be used in the body of a +function definition. More precisely, it can only be used when the closest enclosing *block* is a function definition, not a module-level or class block. (See :ref:`prog_structure` for an explanation of blocks.) @@ -1339,8 +1340,11 @@ For :keyword:`!yield from` expressions, the same restrictions apply. implement comprehensions and generator expressions. -Formal grammar of :keyword:`yield` expressions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Formal grammar of :keyword:`!yield` expressions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The formal grammar of :keyword:`!yield` and :keyword:`!yield from` +expressions is: .. grammar-snippet:: :group: python-grammar @@ -1355,12 +1359,10 @@ Formal grammar of :keyword:`yield` expressions | `expression_list` | `starred_expression` "," [`starred_expression_list`] +Note that ``yield_expression``, the unparenthesized rule, also appears in +:token:`~python-grammar:expression_stmt` and +:token:`~python-grammar:assignment_stmt`. -.. seealso:: - - :pep:`525` - Asynchronous Generators - The proposal that expanded on :pep:`492` by adding generator capabilities to - coroutine functions. .. index:: single: yield; examples @@ -1489,6 +1491,12 @@ of a *finalizer* method see the implementation of The expression ``yield from `` is a syntax error when used in an asynchronous generator function. +.. seealso:: + + :pep:`525` - Asynchronous Generators + The proposal that expanded on :pep:`492` by adding generator capabilities to + coroutine functions. + .. _primaries: Primaries From e79cd8039b8c8e1e79b827c33045f93705bdce11 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 14 Oct 2026 16:14:49 +0200 Subject: [PATCH 10/12] Remove bad edit --- Doc/builtins/stdtypes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/builtins/stdtypes.rst b/Doc/builtins/stdtypes.rst index ed9b9bfffcd995..0f0b1c7b4ce391 100644 --- a/Doc/builtins/stdtypes.rst +++ b/Doc/builtins/stdtypes.rst @@ -1078,7 +1078,7 @@ If a generator iterator is not resumed before it is destroyed raising a :exc:`GeneratorExit` at the :keyword:`!yield` point where the function is suspended. If the :keyword:`!yield` point is in a :keyword:`try` construct, -pending :keyword:`finally` clauses execute. (:keyword:`except GeneratorExit` and +pending :keyword:`finally` clauses execute. .. admonition:: CPython implementation detail From e69cea343c392d10f0a9b041c998f03bff32e096 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 14 Oct 2026 17:19:45 +0200 Subject: [PATCH 11/12] Cleanups --- Doc/builtins/stdtypes.rst | 64 ++++++++++++++++++----------------- Doc/reference/expressions.rst | 46 +++++++++++++------------ 2 files changed, 57 insertions(+), 53 deletions(-) diff --git a/Doc/builtins/stdtypes.rst b/Doc/builtins/stdtypes.rst index 0f0b1c7b4ce391..16d8aec35dbd02 100644 --- a/Doc/builtins/stdtypes.rst +++ b/Doc/builtins/stdtypes.rst @@ -996,7 +996,7 @@ The iterator then allows control over the execution of the underlying function. Initially, the function is *suspended*. Execution starts when a :dfn:`consumer` asks the iterator to produce the next value, for example using the :func:`next` function (which calls -the generator's :meth:`!generator.__next__()` method). +the generator's :meth:`~generator.__next__()` method). The execution starts at the beginning of the function body and proceeds to the first :keyword:`!yield` expression. The expression after the :keyword:`!yield` keyword is evaluated and returned @@ -1015,7 +1015,8 @@ the internal evaluation stack, and the state of any exception handling. When the iterator is asked to produce another value, execution is *resumed*: the function proceeds from the :keyword:`!yield` expression where it was -suspended until the next :keyword:`!yield` expression is encountered:: +suspended and continues until the next :keyword:`!yield` expression +is encountered:: >>> second_value = next(generator_iterator) generator at line 4 @@ -1038,8 +1039,7 @@ a :exc:`StopIteration` exception:: StopIteration Commonly, an iterator is consumed via a :keyword:`for` loop, which implicitly -calls :func:`next` at the start of each iteration, and handles -:exc:`StopIteration`:: +calls :func:`next` repeatedly, and handles :exc:`StopIteration`:: >>> for value in generator_function(): ... print("value from generator:", value) @@ -1049,9 +1049,11 @@ calls :func:`next` at the start of each iteration, and handles value from generator: 456 generator ending -Note that instead of reusing the exhausted ``generator_iterator``, -this example calls ``generator_function()`` again to create a new iterator -with its own local execution state. +.. note:: + + In this example, the ``generator_function()`` call creates a new iterator + with its own local execution state. + The exhausted ``generator_iterator`` from earlier examples is not reused. One common use for generator functions is implementing the :meth:`~object.__iter__` method of custom iterable objects. @@ -1074,10 +1076,11 @@ Generator finalization ^^^^^^^^^^^^^^^^^^^^^^ If a generator iterator is not resumed before it is destroyed -(garbage collected), its :meth:`~generator.close` method will be called, -raising a :exc:`GeneratorExit` at the :keyword:`!yield` point where the +(:term:`garbage collected `), +its :meth:`~generator.close` method will be called, +raising a :exc:`GeneratorExit` at the :keyword:`!yield` expression where the function is suspended. -If the :keyword:`!yield` point is in a :keyword:`try` construct, +If that :keyword:`!yield` expression is in a :keyword:`try` construct, pending :keyword:`finally` clauses execute. .. admonition:: CPython implementation detail @@ -1101,8 +1104,9 @@ pending :keyword:`finally` clauses execute. garbage collection (and thus printing the message) may happen at any later point, or even not at all. -Note that yielding a value after :meth:`~generator.close` raises -:exc:`!GeneratorExit` is an error:: +After :meth:`~generator.close` raises :exc:`!GeneratorExit`, +the underlying function must not yield. +If it does, :exc:`RuntimeError` is raised instead of suspending the function:: >>> def gen(): ... try: @@ -1126,8 +1130,8 @@ Note that yielding a value after :meth:`~generator.close` raises .. index:: pair: object; generator .. _generator-methods: -Generator iterators -^^^^^^^^^^^^^^^^^^^ +Generator iterator methods +^^^^^^^^^^^^^^^^^^^^^^^^^^ Generator iterators implement the :ref:`iterator protocol `. @@ -1139,20 +1143,19 @@ Iterating them drives execution of the underlying generator function. Starts the execution of a generator function or resumes it at the :ref:`yield expression ` where the function is currently suspended. - When a generator function is resumed with a :meth:`~generator.__next__` - method, the current yield expression always evaluates to :const:`None`. - The execution then continues to the next yield expression, where the - generator is suspended again, and the value of the expression after the - :keyword:`yield` keyword is returned to :meth:`~generator.__next__`'s - caller. + The :keyword:`!yield` expression (if any) evaluates to :const:`!None`. + The execution then continues to the next :keyword:`!yield` expression, + where the generator is suspended again, and the value of the expression + after the :keyword:`!yield` keyword is returned to + :meth:`!__next__`'s caller. If the generator exits without yielding another value, - :meth:`~generator.__next__` raises a :exc:`StopIteration` exception, + :meth:`!__next__` raises a :exc:`StopIteration` exception, signalling that iteration has completed. This method is normally called implicitly, for example by a :keyword:`for` loop, or by the built-in :func:`next` function. -Generator iterators have a few more methods than generic iterators, which +Generator iterators have a few more methods than generic iterators. These can be used to control the execution of the underlying generator function: .. method:: generator.send(value) @@ -1164,8 +1167,8 @@ can be used to control the execution of the underlying generator function: the underlying function and either returns the next yielded value or raises :exc:`StopIteration`. - When :meth:`send` is called to start the generator, it must be called - with :const:`None` as the argument, because there is no current yield + When :meth:`!send` is called to start the generator, it must be called + with :const:`!None` as the argument, because there is no current yield expression that could receive the value. @@ -1180,10 +1183,9 @@ can be used to control the execution of the underlying generator function: If the generator function does not catch the passed-in exception, or raises a different exception, then that exception propagates to the caller. - When :meth:`throw` is called to start the generator, the generator - immediately exits (that is, subsequent calls to :meth:`~generator.__next__` - will raise :exc:`StopIteration`) and the thrown exception is propagated to - :meth:`throw`'s caller. + When :meth:`throw` is called to start the generator, the generator is + immediately :term:`exhausted`, the underlying function is not executed, + and the thrown exception is propagated to :meth:`throw`'s caller. In typical use, this is called with a single argument, an exception instance, similar to the way the :keyword:`raise` keyword is used. @@ -1209,9 +1211,9 @@ can be used to control the execution of the underlying generator function: Raises a :exc:`GeneratorExit` exception at the point where the generator function is currently suspended (equivalent to calling ``throw(GeneratorExit)``). - If the generator function has already exited (due to an exception or - normal return), or raises :exc:`GeneratorExit` (by not catching the - exception), :meth:`close` returns :const:`None`. + Then, if the generator function has already exited (due to an exception or + normal return), or raises :exc:`GeneratorExit` (including by not catching + the exception), :meth:`close` returns :const:`None`. If the generator yields a value, a :exc:`RuntimeError` is raised. If the generator raises any other exception, it is propagated to the caller. If a generator returns a value upon being closed, that value is returned diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index e121040ce7fb9e..5ee5edd879bb2d 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -1095,7 +1095,7 @@ see :ref:`yield-expression-placement` for details. A function that contains one or more yield expressions is a :term:`generator function`. -At runtime, calling such a function does not run its code directly, but instead +At runtime, calling such a function does not run its code immediately, but instead returns a :term:`generator iterator` which controls execution of the underlying function. @@ -1122,7 +1122,7 @@ For example:: got: 3 See :ref:`generator-types` for details on runtime behavior and -ways to control execution of the underlying function. +ways to control the execution of the underlying function. If the underlying function is defined using ``async def``, it becomes an :term:`asynchronous generator function` instead. @@ -1157,8 +1157,8 @@ Using :keyword:`!yield` as an expression Usually, :keyword:`!yield` expressions are used alone on a line, as in the examples above. -However, like other expressions, they can also be used in assignments and -more complex expressions. +However, like other expressions, they can also be used in other contexts, +such as assignments and as parts of more complex expressions. .. seealso:: @@ -1176,7 +1176,7 @@ can be used in assignments:: ... print("value of yield is", value) At runtime, when using the basic iteration protocol (:keyword:`for` or -:func:`next`), the value of a :keyword:`yield` expression is ``None``:: +:func:`next`), the value of a :keyword:`!yield` expression is ``None``:: >>> for value in generator_function(): ... pass @@ -1205,6 +1205,7 @@ For example:: >>> def adding_generator(): ... sum_of_yields = (yield "first number") + (yield "second number") ... print("The sum is:", sum_of_yields) + >>> iterator = adding_generator() >>> next(iterator) "first number" @@ -1216,7 +1217,9 @@ For example:: Traceback (most recent call last): StopIteration -More precisely, parentheses can be only omitted when the :keyword:`yield` +Parentheses around :keyword:`yield` expressions are also required in other +contexts, such as :keyword:`if` statement conditions. +They can be only omitted when the :keyword:`yield` expression is: - an :ref:`expression statement `'s only expression, or @@ -1236,7 +1239,7 @@ a :keyword:`!yield from` expression. Like a :keyword:`yield` expression, a :keyword:`!yield from` expression is only allowed in functions and turns the enclosing function into a generator. It must be parenthesized in some contexts; :ref:`the rules ` -are the same as for :keyword:`!yield` expressions. +for parentheses are the same as for :keyword:`!yield` expressions. At runtime, *subiterable* -- the expression after the :keyword:`!yield from` keywords -- must evaluate to an iterable. @@ -1314,24 +1317,27 @@ is a function definition, not a module-level or class block. For example:: class C: - # yield is not allowed here + # yield is not allowed here - def foo(self): - yield 1 # this makes `foo` a generator function + def foo(self): + yield 1 # this makes `foo` a generator function - class NestedClass: + class NestedClass: # yield is not allowed here pass - def nested_function(): + def nested_function(): # this function is *not* a generator def double_nested(): - yield 2 # this makes `double_nested` a generator function + # this `yield` makes `double_nested` (but + # not `nested_function`) a generator function + yield 2 Due to their side effects on the containing scope, ``yield`` expressions are not permitted as part of the implicitly defined scopes used to -implement comprehensions and generator expressions. +implement :ref:`comprehensions ` and +:ref:`generator expressions `. For :keyword:`!yield from` expressions, the same restrictions apply. @@ -1353,11 +1359,7 @@ expressions is: yield_expression: | 'yield' 'from' `expression` - | 'yield' [`yield_list`] - - yield_list: - | `expression_list` - | `starred_expression` "," [`starred_expression_list`] + | 'yield' [`starred_expression_list`] Note that ``yield_expression``, the unparenthesized rule, also appears in :token:`~python-grammar:expression_stmt` and @@ -1366,8 +1368,8 @@ Note that ``yield_expression``, the unparenthesized rule, also appears in .. index:: single: yield; examples -Examples -^^^^^^^^ +Examples of :keyword:`!yield` expressions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Here is a simple example that demonstrates the behavior of generators and generator functions:: @@ -1446,7 +1448,7 @@ how a generator object would be used in a :keyword:`for` statement. Calling one of the asynchronous generator's methods returns an :term:`awaitable` object, and the execution starts when this object is awaited on. At that time, the execution proceeds to the first yield expression, where it is suspended -again, returning the value of :token:`~python-grammar:yield_list` to the +again, returning the value of the expression after :keyword:`!yield` to the awaiting coroutine. As with a generator, suspension means that all local state is retained, including the current bindings of local variables, the instruction pointer, the internal evaluation stack, and the state of any exception handling. From 8ae03064c21c9ea817d8d41282c512f3e68c224a Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 14 Oct 2026 17:22:52 +0200 Subject: [PATCH 12/12] Fixups --- Doc/reference/simple_stmts.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index 750539bb8a50a9..74d71da9e68d51 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -73,7 +73,7 @@ The formal grammar for an expression statement is: .. note:: - The formal grammar includes as special case for a single :keyword:`!yield` + The formal grammar includes a special case for a single :keyword:`!yield` expression, without parentheses around it. This is sometimes called a :dfn:`yield statement`. For complete documentation of :keyword:`!yield` semantics, see