gh-141984: Reword the "yield expressions"` section

Doc/builtins/stdtypes.rst

@@ -976,32 +976,84 @@ A function that contains one or more :ref:`yield expressions <yieldexpr>`
 is a :term:`generator function`.
 For example::
 
-   >>> def count_to_three():
-   ...     yield 0
-   ...     yield 1
-   ...     yield 2
-   ...     yield 3
+   >>> def generator_function():
+   ...     print("generator starting")
+   ...     yield 123
+   ...     print("generator at line 4")
+   ...     yield 456
+   ...     print("generator ending")
 
 Generator functions behave as regular
 :ref:`user-defined functions <user-defined-funcs>`
 (for example, they have the same attributes), except that calling a generator
-function returns a :ref:`generator iterator <generator-methods>`::
+function returns a :term:`generator iterator`::
+
+   >>> generator_iterator = generator_function()
+   >>> generator_iterator
+   <generator object generator_function at ...>
+
+The iterator then allows control over the execution of the underlying function.
+Initially, the function is *suspended*.
+Execution starts when a :dfn:`consumer` asks the iterator to produce the next
+value, for example using the :func:`next` function (which calls
+the generator's :meth:`~generator.__next__()` method).
+The execution starts at the beginning of the function body
+and proceeds to the first :keyword:`!yield` expression.
+The expression after the :keyword:`!yield` keyword is evaluated and returned
+to the consumer (here, the caller of ``next``)::
+
+   >>> first_value = next(generator_iterator)
+   generator starting
+   >>> print(first_value)
+   123
+
+At this point, execution of the generator function is *suspended* again.
+By *suspended*, we mean that all local execution state is retained, including the
+current bindings of local variables, the "instruction pointer" (index of
+the :keyword:`!yield` expression at which execution is suspended),
+the internal evaluation stack, and the state of any exception handling.
+
+When the iterator is asked to produce another value, execution is *resumed*:
+the function proceeds from the :keyword:`!yield` expression where it was
+suspended and continues until the next :keyword:`!yield` expression
+is encountered::
+
+   >>> second_value = next(generator_iterator)
+   generator at line 4
+   >>> print(second_value)
+   456
+
+If the generator function returns before a :keyword:`!yield` expression is
+encountered (either with a :keyword:`return` statement, or implicitly at the
+end of the function), the generator iterator becomes :term:`exhausted`.
+All further attempts to get the next value from it will raise
+a :exc:`StopIteration` exception::
+
+   >>> third_value = next(generator_iterator)
+   Traceback (most recent call last):
+     ...
+   StopIteration
+   >>> third_value = next(generator_iterator)
+   Traceback (most recent call last):
+     ...
+   StopIteration
 
-   >>> count_to_three()
-   <generator object count_to_three at 0x7f33a2305000>
+Commonly, an iterator is consumed via a :keyword:`for` loop, which implicitly
+calls :func:`next` repeatedly, and handles :exc:`StopIteration`::
 
-Iterating a generator iterator executes code of the underlying
-generator function, producing each :keyword:`yield`\ed value in turn::
+   >>> for value in generator_function():
+   ...     print("value from generator:", value)
+   generator starting
+   value from generator: 123
+   generator at line 4
+   value from generator: 456
+   generator ending
 
-   >>> for number in count_to_three():
-   ...     print(number)
-   0
-   1
-   2
-   3
+.. note::
 
-   >>> list(count_to_three())
-   [0, 1, 2, 3]
+   In this example, the ``generator_function()`` call creates a new iterator
+   with its own local execution state.
+   The exhausted ``generator_iterator`` from earlier examples is not reused.
 
 One common use for generator functions is implementing the
 :meth:`~object.__iter__` method of custom iterable objects.
@@ -1015,12 +1067,71 @@ For example::
    >>> list(CardDeck())
    ['three of clubs', 'ace of hearts']
 
+.. seealso::
+
+   :pep:`255` - Simple Generators
+      The proposal for adding generators and the :keyword:`yield` statement to Python.
+
+Generator finalization
+^^^^^^^^^^^^^^^^^^^^^^
+
+If a generator iterator is not resumed before it is destroyed
+(:term:`garbage collected <garbage collection>`),
+its :meth:`~generator.close` method will be called,
+raising a :exc:`GeneratorExit` at the :keyword:`!yield` expression where the
+function is suspended.
+If that :keyword:`!yield` expression is in a :keyword:`try` construct,
+pending :keyword:`finally` clauses execute.
+
+.. admonition:: CPython implementation detail
+
+   In the following example, ``iterator`` is garbage collected immediately
+   after the ``del iterator`` statement, triggering the ``finally`` block::
+
+      >>> def gen():
+      ...     try:
+      ...         yield 123
+      ...     finally:
+      ...         print('shutting down')
+      ...
+      >>> iterator = gen()
+      >>> next(iterator)
+      123
+      >>> del iterator
+      shutting down
+
+   On non-CPython implementations, or future versions of CPython,
+   garbage collection (and thus printing the message) may happen at any
+   later point, or even not at all.
+
+After :meth:`~generator.close` raises :exc:`!GeneratorExit`,
+the underlying function must not yield.
+If it does, :exc:`RuntimeError` is raised instead of suspending the function::
+
+   >>> def gen():
+   ...     try:
+   ...         yield 123
+   ...     except GeneratorExit:
+   ...         pass  # ignore the exception
+   ...     yield 456
+   ...
+   >>> iterator = gen()
+   >>> next(iterator)
+   123
+   >>> iterator.close()
+   Traceback (most recent call last):
+   File "<python-input-4>", line 1, in <module>
+      iterator.close()
+      ~~~~~~~~^^
+   RuntimeError: generator ignored GeneratorExit
+
+
 
 .. index:: pair: object; generator
 .. _generator-methods:
 
-Generator iterators
-^^^^^^^^^^^^^^^^^^^
+Generator iterator methods
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Generator iterators implement the
 :ref:`iterator protocol <stdtypes-iterators>`.
@@ -1032,20 +1143,19 @@ Iterating them drives execution of the underlying generator function.
 
    Starts the execution of a generator function or resumes it at the
    :ref:`yield expression <yieldexpr>` where the function is currently suspended.
-   When a generator function is resumed with a :meth:`~generator.__next__`
-   method, the current yield expression always evaluates to :const:`None`.
-   The execution then continues to the next yield expression, where the
-   generator is suspended again, and the value of the expression after the
-   :keyword:`yield` keyword is returned to :meth:`~generator.__next__`'s
-   caller.
+   The :keyword:`!yield` expression (if any) evaluates to :const:`!None`.
+   The execution then continues to the next :keyword:`!yield` expression,
+   where the generator is suspended again, and the value of the expression
+   after the :keyword:`!yield` keyword is returned to
+   :meth:`!__next__`'s caller.
    If the generator exits without yielding another value,
-   :meth:`~generator.__next__` raises a :exc:`StopIteration` exception,
+   :meth:`!__next__` raises a :exc:`StopIteration` exception,
    signalling that iteration has completed.
 
    This method is normally called implicitly, for example by a :keyword:`for`
    loop, or by the built-in :func:`next` function.
 
-Generator iterators have a few more methods than generic iterators, which
+Generator iterators have a few more methods than generic iterators. These
 can be used to control the execution of the underlying generator function:
 
 .. method:: generator.send(value)
@@ -1057,8 +1167,8 @@ can be used to control the execution of the underlying generator function:
    the underlying function and either returns the next yielded value or raises
    :exc:`StopIteration`.
 
-   When :meth:`send` is called to start the generator, it must be called
-   with :const:`None` as the argument, because there is no current yield
+   When :meth:`!send` is called to start the generator, it must be called
+   with :const:`!None` as the argument, because there is no current yield
    expression that could receive the value.
 
 
@@ -1073,10 +1183,9 @@ can be used to control the execution of the underlying generator function:
    If the generator function does not catch the passed-in exception, or
    raises a different exception, then that exception propagates to the caller.
 
-   When :meth:`throw` is called to start the generator, the generator
-   immediately exits (that is, subsequent calls to :meth:`~generator.__next__`
-   will raise :exc:`StopIteration`) and the thrown exception is propagated to
-   :meth:`throw`'s caller.
+   When :meth:`throw` is called to start the generator, the generator is
+   immediately :term:`exhausted`, the underlying function is not executed,
+   and the thrown exception is propagated to :meth:`throw`'s caller.
 
    In typical use, this is called with a single argument, an exception instance,
    similar to the way the :keyword:`raise` keyword is used.
@@ -1102,9 +1211,9 @@ can be used to control the execution of the underlying generator function:
    Raises a :exc:`GeneratorExit` exception at the point where the generator
    function is currently suspended (equivalent to calling ``throw(GeneratorExit)``).
 
-   If the generator function has already exited (due to an exception or
-   normal return), or raises :exc:`GeneratorExit` (by not catching the
-   exception), :meth:`close` returns :const:`None`.
+   Then, if the generator function has already exited (due to an exception or
+   normal return), or raises :exc:`GeneratorExit` (including by not catching
+   the exception), :meth:`close` returns :const:`None`.
    If the generator yields a value, a :exc:`RuntimeError` is raised.
    If the generator raises any other exception, it is propagated to the caller.
    If a generator returns a value upon being closed, that value is returned
@@ -1125,6 +1234,12 @@ Calling any of the generator methods (:meth:`~generator.__next__`,
 while one of these methods is already executing
 raises a :exc:`ValueError` exception.
 
+.. seealso::
+
+   :pep:`342` - Coroutines via Enhanced Generators
+      The proposal to add :meth:`!send`, :meth:`!throw` and :meth:`!close`,
+      making generators usable as simple coroutines.
+
 
 .. index:: pair: object; asynchronous-generator
 .. _asynchronous-generator-methods:

Doc/glossary.rst

@@ -649,15 +649,15 @@ Glossary
       collector that is able to detect and break reference cycles.  The
       garbage collector can be controlled using the :mod:`gc` module.
 
-      .. index:: single: generator
+   .. index:: single: generator
 
    generator
       Informally used to mean either a :term:`generator function` or a
       :term:`generator iterator`, depending on context.  The formal terms
       :term:`generator function` and :term:`generator iterator` are uncommon
       in practice; "generator" alone is almost always sufficient.
 
-      .. index:: single: generator function
+   .. index:: single: generator function
 
    generator function
       A function which returns a :term:`generator` object.  It looks like a
@@ -680,7 +680,7 @@ Glossary
       :meth:`~generator.throw` method to raise an exception at the point
       where the generator was paused.  See :ref:`generator-methods`.
 
-      .. index:: single: generator expression
+   .. index:: single: generator expression
 
    generator expression
       An :term:`expression` that returns an :term:`iterator`.  It looks like a normal expression