bpo-10544: Deprecate "yield" in comprehensions and generator expressions. (GH-4579)

The current behaviour of yield expressions inside comprehensions  and
generator expressions is essentially an accident of implementation - it
arises implicitly from the way the compiler handles yield expressions inside
nested functions and generators.

Since the current behaviour wasn't deliberately designed, and is inherently
confusing, we're deprecating it, with no current plans to reintroduce it.
Instead, our advice will be to use a named nested generator definition
for cases where this behaviour is desired.

6 files changed, 134 insertions, 20 deletions

diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst
index 1cff8a52df..fb92ad0f07 100644
--- a/Doc/reference/expressions.rst
+++ b/Doc/reference/expressions.rst
@@ -183,8 +183,21 @@ by considering each of the :keyword:`for` or :keyword:`if` clauses a block,
 nesting from left to right, and evaluating the expression to produce an element
 each time the innermost block is reached.
 
-Note that the comprehension is executed in a separate scope, so names assigned
-to in the target list don't "leak" into the enclosing scope.
+However, aside from the iterable expression in the leftmost :keyword:`for` clause,
+the comprehension is executed in a separate implicitly nested scope. This ensures
+that names assigned to in the target list don't "leak" into the enclosing scope.
+
+The iterable expression in the leftmost :keyword:`for` clause is evaluated
+directly in the enclosing scope and then passed as an argument to the implictly
+nested scope. Subsequent :keyword:`for` clauses and any filter condition in the
+leftmost :keyword:`for` clause cannot be evaluated in the enclosing scope as
+they may depend on the values obtained from the leftmost iterable. For example:
+``[x*y for x in range(10) for y in range(x, x+10)]``.
+
+To ensure the comprehension always results in a container of the appropriate
+type, ``yield`` and ``yield from`` expressions are prohibited in the implicitly
+nested scope (in Python 3.7, such expressions emit :exc:`DeprecationWarning`
+when compiled, in Python 3.8+ they will emit :exc:`SyntaxError`).
 
 Since Python 3.6, in an :keyword:`async def` function, an :keyword:`async for`
 clause may be used to iterate over a :term:`asynchronous iterator`.
@@ -198,6 +211,13 @@ or :keyword:`await` expressions it is called an
 suspend the execution of the coroutine function in which it appears.
 See also :pep:`530`.
 
+.. versionadded:: 3.6
+   Asynchronous comprehensions were introduced.
+
+.. deprecated:: 3.7
+   ``yield`` and ``yield from`` deprecated in the implicitly nested scope.
+
+
 .. _lists:
 
 List displays
@@ -316,27 +336,42 @@ brackets or curly braces.
 
 Variables used in the generator expression are evaluated lazily when the
 :meth:`~generator.__next__` method is called for the generator object (in the same
-fashion as normal generators).  However, the leftmost :keyword:`for` clause is
-immediately evaluated, so that an error produced by it can be seen before any
-other possible error in the code that handles the generator expression.
-Subsequent :keyword:`for` clauses cannot be evaluated immediately since they
-may depend on the previous :keyword:`for` loop. For example: ``(x*y for x in
-range(10) for y in bar(x))``.
+fashion as normal generators).  However, the iterable expression in the
+leftmost :keyword:`for` clause is immediately evaluated, so that an error
+produced by it will be emitted at the point where the generator expression
+is defined, rather than at the point where the first value is retrieved.
+Subsequent :keyword:`for` clauses and any filter condition in the leftmost
+:keyword:`for` clause cannot be evaluated in the enclosing scope as they may
+depend on the values obtained from the leftmost iterable. For example:
+``(x*y for x in range(10) for y in range(x, x+10))``.
 
 The parentheses can be omitted on calls with only one argument.  See section
 :ref:`calls` for details.
 
+To avoid interfering with the expected operation of the generator expression
+itself, ``yield`` and ``yield from`` expressions are prohibited in the
+implicitly defined generator (in Python 3.7, such expressions emit
+:exc:`DeprecationWarning` when compiled, in Python 3.8+ they will emit
+:exc:`SyntaxError`).
+
 If a generator expression contains either :keyword:`async for`
 clauses or :keyword:`await` expressions it is called an
 :dfn:`asynchronous generator expression`.  An asynchronous generator
 expression returns a new asynchronous generator object,
 which is an asynchronous iterator (see :ref:`async-iterators`).
 
+.. versionadded:: 3.6
+   Asynchronous generator expressions were introduced.
+
 .. versionchanged:: 3.7
    Prior to Python 3.7, asynchronous generator expressions could
    only appear in :keyword:`async def` coroutines.  Starting
    with 3.7, any function can use asynchronous generator expressions.
 
+.. deprecated:: 3.7
+   ``yield`` and ``yield from`` deprecated in the implicitly nested scope.
+
+
 .. _yieldexpr:
 
 Yield expressions
@@ -364,6 +399,16 @@ coroutine function to be an asynchronous generator. For example::
     async def agen(): # defines an asynchronous generator function (PEP 525)
         yield 123
 
+Due to their side effects on the containing scope, ``yield`` expressions
+are not permitted as part of the implicitly defined scopes used to
+implement comprehensions and generator expressions (in Python 3.7, such
+expressions emit :exc:`DeprecationWarning` when compiled, in Python 3.8+
+they will emit :exc:`SyntaxError`)..
+
+.. deprecated:: 3.7
+   Yield expressions deprecated in the implicitly nested scopes used to
+   implement comprehensions and generator expressions.
+
 Generator functions are described below, while asynchronous generator
 functions are described separately in section
 :ref:`asynchronous-generator-functions`.
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 6545a18a99..b6dad4eab6 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -570,6 +570,18 @@ Other CPython Implementation Changes
 Deprecated
 ==========
 
+* Yield expressions (both ``yield`` and ``yield from`` clauses) are now deprecated
+  in comprehensions and generator expressions (aside from the iterable expression
+  in the leftmost :keyword:`for` clause). This ensures that comprehensions
+  always immediately return a container of the appropriate type (rather than
+  potentially returning a :term:`generator iterator` object), while generator
+  expressions won't attempt to interleave their implicit output with the output
+  from any explicit yield expressions.
+
+  In Python 3.7, such expressions emit :exc:`DeprecationWarning` when compiled,
+  in Python 3.8+ they will emit :exc:`SyntaxError`. (Contributed by Serhiy
+  Storchaka in :issue:`10544`.)
+
 - Function :c:func:`PySlice_GetIndicesEx` is deprecated and replaced with
   a macro if ``Py_LIMITED_API`` is not set or set to the value between
   ``0x03050400`` and ``0x03060000`` (not including) or ``0x03060100`` or
diff --git a/Lib/test/test_generators.py b/Lib/test/test_generators.py
index 7eac9d076b..f88c762581 100644
--- a/Lib/test/test_generators.py
+++ b/Lib/test/test_generators.py
@@ -1830,13 +1830,7 @@ Yield by itself yields None:
 [None]
 
 
-
-An obscene abuse of a yield expression within a generator expression:
-
->>> list((yield 21) for i in range(4))
-[21, None, 21, None, 21, None, 21, None]
-
-And a more sane, but still weird usage:
+Yield is allowed only in the outermost iterable in generator expression:
 
 >>> def f(): list(i for i in [(yield 26)])
 >>> type(f())
@@ -2106,10 +2100,6 @@ enclosing function a generator:
 >>> type(f())
 <class 'generator'>
 
->>> def f(): x=(i for i in (yield) if (yield))
->>> type(f())
-<class 'generator'>
-
 >>> def f(d): d[(yield "a")] = d[(yield "b")] = 27
 >>> data = [1,2]
 >>> g = f(data)
diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py
index 65e26bfd38..823315f8cd 100644
--- a/Lib/test/test_grammar.py
+++ b/Lib/test/test_grammar.py
@@ -841,6 +841,41 @@ class GrammarTests(unittest.TestCase):
         # Check annotation refleak on SyntaxError
         check_syntax_error(self, "def g(a:(yield)): pass")
 
+    def test_yield_in_comprehensions(self):
+        # Check yield in comprehensions
+        def g(): [x for x in [(yield 1)]]
+        def g(): [x for x in [(yield from ())]]
+
+        def check(code, warntext):
+            with self.assertWarnsRegex(DeprecationWarning, warntext):
+                compile(code, '<test string>', 'exec')
+            import warnings
+            with warnings.catch_warnings():
+                warnings.filterwarnings('error', category=DeprecationWarning)
+                with self.assertRaisesRegex(SyntaxError, warntext):
+                    compile(code, '<test string>', 'exec')
+
+        check("def g(): [(yield x) for x in ()]",
+              "'yield' inside list comprehension")
+        check("def g(): [x for x in () if not (yield x)]",
+              "'yield' inside list comprehension")
+        check("def g(): [y for x in () for y in [(yield x)]]",
+              "'yield' inside list comprehension")
+        check("def g(): {(yield x) for x in ()}",
+              "'yield' inside set comprehension")
+        check("def g(): {(yield x): x for x in ()}",
+              "'yield' inside dict comprehension")
+        check("def g(): {x: (yield x) for x in ()}",
+              "'yield' inside dict comprehension")
+        check("def g(): ((yield x) for x in ())",
+              "'yield' inside generator expression")
+        check("def g(): [(yield from x) for x in ()]",
+              "'yield' inside list comprehension")
+        check("class C: [(yield x) for x in ()]",
+              "'yield' inside list comprehension")
+        check("[(yield x) for x in ()]",
+              "'yield' inside list comprehension")
+
     def test_raise(self):
         # 'raise' test [',' test]
         try: raise RuntimeError('just testing')
diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-11-27-08-37-34.bpo-10544.07nioT.rst b/Misc/NEWS.d/next/Core and Builtins/2017-11-27-08-37-34.bpo-10544.07nioT.rst
new file mode 100644
index 0000000000..555c94ebae
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2017-11-27-08-37-34.bpo-10544.07nioT.rst
@@ -0,0 +1,3 @@
+Yield expressions are now deprecated in comprehensions and generator
+expressions. They are still permitted in the definition of the outermost
+iterable, as that is evaluated directly in the enclosing scope.
diff --git a/Python/symtable.c b/Python/symtable.c
index 55815c91cc..bbac25cf37 100644
--- a/Python/symtable.c
+++ b/Python/symtable.c
@@ -1734,7 +1734,6 @@ symtable_handle_comprehension(struct symtable *st, expr_ty e,
                               e->lineno, e->col_offset)) {
         return 0;
     }
-    st->st_cur->ste_generator = is_generator;
     if (outermost->is_async) {
         st->st_cur->ste_coroutine = 1;
     }
@@ -1754,6 +1753,36 @@ symtable_handle_comprehension(struct symtable *st, expr_ty e,
     if (value)
         VISIT(st, expr, value);
     VISIT(st, expr, elt);
+    if (st->st_cur->ste_generator) {
+        PyObject *msg = PyUnicode_FromString(
+            (e->kind == ListComp_kind) ? "'yield' inside list comprehension" :
+            (e->kind == SetComp_kind) ? "'yield' inside set comprehension" :
+            (e->kind == DictComp_kind) ? "'yield' inside dict comprehension" :
+            "'yield' inside generator expression");
+        if (msg == NULL) {
+            symtable_exit_block(st, (void *)e);
+            return 0;
+        }
+        if (PyErr_WarnExplicitObject(PyExc_DeprecationWarning,
+                msg, st->st_filename, st->st_cur->ste_lineno,
+                NULL, NULL) == -1)
+        {
+            if (PyErr_ExceptionMatches(PyExc_DeprecationWarning)) {
+                /* Replace the DeprecationWarning exception with a SyntaxError
+                   to get a more accurate error report */
+                PyErr_Clear();
+                PyErr_SetObject(PyExc_SyntaxError, msg);
+                PyErr_SyntaxLocationObject(st->st_filename,
+                                           st->st_cur->ste_lineno,
+                                           st->st_cur->ste_col_offset);
+            }
+            Py_DECREF(msg);
+            symtable_exit_block(st, (void *)e);
+            return 0;
+        }
+        Py_DECREF(msg);
+    }
+    st->st_cur->ste_generator |= is_generator;
     return symtable_exit_block(st, (void *)e);
 }