4. Python 中的协程¶
4. Coroutines in Python
Python 提供了一流的协程，具有 “coroutine” 类型和新表达式，例如 “async def” 和 “await”。
它提供了“asyncio”模块来运行协程和开发异步程序。
在本节中，我们将更仔细地了解协程。
Python provides first-class coroutines with a “coroutine” type and new expressions like “async def” and “await“.
It provides the “asyncio” module for running coroutines and developing asynchronous programs.
In this section, we will take a much closer look at coroutines.
4.1 什么是协程¶
4.1 What is a Coroutine
协程是一个可以暂停和恢复的函数。
它通常被定义为广义子例程。
可以执行子例程，从一点开始并在另一点结束。 然而，协程可以执行然后挂起，并在最终终止之前恢复多次。
具体来说，协程可以控制何时暂停执行。
这可能涉及特定表达式的使用，例如Python中的“await”表达式，就像Python生成器中的yield表达式一样。
协程是一种可以在我们有可能长时间运行的任务时暂停并在该任务完成时恢复的方法。 在Python 3.5版本中，当关键字async和await被显式添加到语言中时，该语言实现了对协程和异步编程的一流支持。
— PAGE 3, PYTHON CONCURRENCY WITH ASYNCIO, 2022.
协程可能会因多种原因而挂起，例如执行另一个协程，例如 等待另一个任务，或者等待一些外部资源，例如套接字连接或进程返回数据。
协程用于并发。
协程让您可以在 Python 程序中拥有大量看似同时存在的函数。
— PAGE 267, EFFECTIVE PYTHON, 2019.
可以同时创建和执行许多协程。 它们可以控制何时挂起和恢复，从而允许它们在执行并发任务时进行合作。
这称为“协作多任务处理”(https://en.wikipedia.org/wiki/Cooperative_multitasking)，与通常与线程一起使用的多任务处理（称为抢占式多任务处理）不同。
… 为了同时运行多个应用程序，进程会定期或在空闲或逻辑阻塞时自愿放弃控制权。 这种类型的多任务处理称为协作，因为所有程序都必须协作才能使调度方案发挥作用。
抢占式多任务处理涉及操作系统选择要挂起和恢复的线程以及何时执行，而不是在协作多任务处理的情况下由任务本身决定。
现在我们已经了解了什么是协程，让我们通过将它们与其他熟悉的编程结构进行比较来加深理解。
A coroutine is a function that can be suspended and resumed.
It is often defined as a generalized subroutine.
A subroutine can be executed, starting at one point and finishing at another point. Whereas, a coroutine can be executed then suspended, and resumed many times before finally terminating.
Specifically, coroutines have control over when exactly they suspend their execution.
This may involve the use of a specific expression, such as an “await” expression in Python, like a yield expression in a Python generator.
A coroutine is a method that can be paused when we have a potentially long-running task and then resumed when that task is finished. In Python version 3.5, the language implemented first-class support for coroutines and asynchronous programming when the keywords async and await were explicitly added to the language.
— PAGE 3, PYTHON CONCURRENCY WITH ASYNCIO, 2022.
A coroutine may suspend for many reasons, such as executing another coroutine, e.g. awaiting another task, or waiting for some external resources, such as a socket connection or process to return data.
Coroutines are used for concurrency.
Coroutines let you have a very large number of seemingly simultaneous functions in your Python programs.
— PAGE 267, EFFECTIVE PYTHON, 2019.
Many coroutines can be created and executed at the same time. They have control over when they will suspend and resume, allowing them to cooperate as to when concurrent tasks are executed.
This is called cooperative multitasking and is different from the multitasking typically used with threads called preemptive multitasking tasking.
… in order to run multiple applications concurrently, processes voluntarily yield control periodically or when idle or logically blocked. This type of multitasking is called cooperative because all programs must cooperate for the scheduling scheme to work.
Preemptive multitasking involves the operating system choosing what threads to suspend and resume and when to do so, as opposed to the tasks themselves deciding in the case of cooperative multitasking.
Now that we have some idea of what a coroutine is, let’s deepen this understanding by comparing them to other familiar programming constructs.
4.2 协程与例程和子例程¶
4.2 Coroutine vs Routine and Subroutine
在现代编程中，“例程”和“子例程”通常指同一事物。
也许更正确的是，例程是一个程序，而子例程是程序中的一个函数。
例程有子例程。
它是一个离散的表达式模块，被分配了一个名称，可以接受参数并且可以返回一个值。
- 子例程(Subroutine): A module of instructions that can be executed on demand, typically named, and may take arguments and return a value. also called a function
执行子例程，运行表达式，然后以某种方式返回。 通常，一个子例程被另一个子例程调用。
协程是子程序的扩展。 这意味着子例程是一种特殊类型的协程。
协程在很多方面都类似于子例程，例如：
- 它们都是离散的命名表达式模块。
- 他们都可以接受争论，也可以不接受争论。
- 它们都可以返回值，也可以不返回值。
主要区别在于，它在返回和退出之前多次选择暂停和恢复执行。
协程和子例程都可以调用它们自己的其他示例。 一个子程序可以调用其他子程序。 协程执行其他协程。 然而，协程也可以执行其他子例程。
当一个协程执行另一个协程时，它必须暂停执行，并在另一个协程完成后允许另一个协程恢复。
这就像一个子例程调用另一个子例程。 不同之处在于协程的挂起可能允许任意数量的其他协程运行。
这使得调用另一个协程的协程比调用另一个子例程的子例程更强大。 它是协程促进的协作多任务处理的核心。
A “routine” and “subroutine” often refer to the same thing in modern programming.
Perhaps more correctly, a routine is a program, whereas a subroutine is a function in the program.
A routine has subroutines.
It is a discrete module of expressions that is assigned a name, may take arguments and may return a value.
- Subroutine: A module of instructions that can be executed on demand, typically named, and may take arguments and return a value. also called a function
A subroutine is executed, runs through the expressions, and returns somehow. Typically, a subroutine is called by another subroutine.
A coroutine is an extension of a subroutine. This means that a subroutine is a special type of a coroutine.
A coroutine is like a subroutine in many ways, such as:
- They both are discrete named modules of expressions.
- They both can take arguments, or not.
- They both can return a value, or not.
The main difference is that it chooses to suspend and resume its execution many times before returning and exiting.
Both coroutines and subroutines can call other examples of themselves. A subroutine can call other subroutines. A coroutine executes other coroutines. However, a coroutine can also execute other subroutines.
When a coroutine executes another coroutine, it must suspend its execution and allow the other coroutine to resume once the other coroutine has completed.
This is like a subroutine calling another subroutine. The difference is the suspension of the coroutine may allow any number of other coroutines to run as well.
This makes a coroutine calling another coroutine more powerful than a subroutine calling another subroutine. It is central to the cooperating multitasking facilitated by coroutines.
4.3 协程与生成器¶
4.3 Coroutine vs Generator
生成器是一个可以暂停其执行的特殊函数。
generator: 返回生成器迭代器的函数。 它看起来像一个普通函数，只不过它包含用于生成一系列可在 for 循环中使用的值的yield 表达式，或者可以使用 next() 函数一次检索一个值。
生成器函数可以像普通函数一样定义，尽管它在暂停执行并返回值时使用了yield 表达式。
生成器函数将返回一个可以遍历的生成器迭代器对象，例如通过 for 循环。 每次执行生成器时，它都会从最后一个暂停点运行到下一个yield 语句。
generator iterator: 由生成器函数创建的对象。 每个yield都会暂时挂起处理，记住位置执行状态（包括局部变量和挂起的try语句）。 当生成器迭代器恢复时，它会从上次停止的地方继续（与每次调用时重新开始的函数相反）。
协程可以使用 “await” 表达式挂起或屈服于另一个协程。 一旦等待的协程完成，它将从此时开始恢复。
使用这种范例，await 语句在功能上类似于yield 语句； 当其他代码运行时，当前函数的执行会暂停。 一旦 await 或 yield 解析出数据，该函数就会恢复。
— PAGE 218, HIGH PERFORMANCE PYTHON, 2020.
我们可以将生成器视为循环中使用的特殊类型的协程和协作多任务处理。
生成器，也称为半协程，是协程的子集。
在开发协程之前，生成器已被扩展，以便它们可以像 Python 程序中的协程一样使用。
这需要大量的生成器技术知识和自定义任务调度程序的开发。
要使用生成器实现您自己的并发，您首先需要对生成器函数和yield 语句有基本的了解。 具体来说，yield 的基本行为是它导致生成器暂停其执行。 通过暂停执行，可以编写一个调度程序，将生成器视为一种“任务”，并使用一种协作任务切换来交替执行它们。
— PAGE 524, PYTHON COOKBOOK, 2013.
这是通过对生成器的更改和引入 “yield from” 表达式实现的。
这些后来被弃用，取而代之的是现代的 async/await 表达式。
A generator is a special function that can suspend its execution.
generator: A function which returns a generator iterator. It looks like a normal function except that it contains yield expressions for producing a series of values usable in a for-loop or that can be retrieved one at a time with the next() function.
A generator function can be defined like a normal function although it uses a yield expression at the point it will suspend its execution and return a value.
A generator function will return a generator iterator object that can be traversed, such as via a for-loop. Each time the generator is executed, it runs from the last point it was suspended to the next yield statement.
generator iterator: An object created by a generator function. Each yield temporarily suspends processing, remembering the location execution state (including local variables and pending try-statements). When the generator iterator resumes, it picks up where it left off (in contrast to functions which start fresh on every invocation).
A coroutine can suspend or yield to another coroutine using an “await” expression. It will then resume from this point once the awaited coroutine has been completed.
Using this paradigm, an await statement is similar in function to a yield statement; the execution of the current function gets paused while other code is run. Once the await or yield resolves with data, the function is resumed.
— PAGE 218, HIGH PERFORMANCE PYTHON, 2020.
We might think of a generator as a special type of coroutine and cooperative multitasking used in loops.
Generators, also known as semicoroutines, are a subset of coroutines.
Before coroutines were developed, generators were extended so that they might be used like coroutines in Python programs.
This required a lot of technical knowledge of generators and the development of custom task schedulers.
To implement your own concurrency using generators, you first need a fundamental insight concerning generator functions and the yield statement. Specifically, the fundamental behavior of yield is that it causes a generator to suspend its execution. By suspending execution, it is possible to write a scheduler that treats generators as a kind of “task” and alternates their execution using a kind of cooperative task switching.
— PAGE 524, PYTHON COOKBOOK, 2013.
This was made possible via changes to the generators and the introduction of the “yield from” expression.
These were later deprecated in favor of the modern async/await expressions.
4.4 协程与任务¶
4.4 Coroutine vs Task
子例程和协程可以代表程序中的“任务”。
但是，在 Python 中，有一个称为 asyncio.Task 对象的特定对象。
运行 Python 协程的类似 Future 的对象。 [...] 任务用于在事件循环中运行协程。
— ASYNCIO TASK OBJECT
协程可以包装在 asyncio.Task 对象中并独立执行，而不是直接在协程内执行。 Task 对象提供异步执行协程的句柄。
- Task: 可以独立执行的包装协程。
这允许包装的协程在后台执行。 调用协程可以继续执行指令，而不是等待另一个协程。
任务不能单独存在，它必须包装一个协程。
因此，任务是协程，但协程不是任务。
您可以在教程中了解有关 asyncio.Task 对象的更多信息：
A subroutine and a coroutine may represent a “task” in a program.
However, in Python, there is a specific object called an asyncio.Task object.
A Future-like object that runs a Python coroutine. […] Tasks are used to run coroutines in event loops.
— ASYNCIO TASK OBJECT
A coroutine can be wrapped in an asyncio.Task object and executed independently, as opposed to being executed directly within a coroutine. The Task object provides a handle on the asynchronously execute coroutine.
- Task: A wrapped coroutine that can be executed independently.
This allows the wrapped coroutine to execute in the background. The calling coroutine can continue executing instructions rather than awaiting another coroutine.
A Task cannot exist on its own, it must wrap a coroutine.
Therefore a Task is a coroutine, but a coroutine is not a task.
You can learn more about asyncio.Task objects in the tutorial:
4.5 协程与线程¶
4.5 Coroutine vs Thread
协程比线程更轻量。
- Thread: 与协程相比重量级
- Coroutine: 与线程相比轻量级。
协程被定义为一个函数。
线程是由底层操作系统创建和管理的对象，在 Python 中表示为 threading.Thread 对象。
- Thread: 由操作系统管理，由Python对象表示。
这意味着协程的创建和开始执行速度通常更快，并且占用的内存更少。 相反，线程的创建和启动速度比协程慢，并且占用更多内存。
启动协程的成本是函数调用。 一旦协程处于活动状态，它就会使用不到 1 KB 的内存，直到耗尽为止。
— PAGE 267, EFFECTIVE PYTHON, 2019.
协程在一个线程内执行，因此单个线程可以执行多个协程。
许多单独的异步函数似乎都是同时运行的，模仿了 Python 线程的并发行为。 然而，协程可以做到这一点，无需内存开销、启动和上下文切换成本，也无需线程所需的复杂锁定和同步代码。
— PAGE 267, EFFECTIVE PYTHON, 2019.
您可以在指南中了解有关线程的更多信息：
A coroutine is more lightweight than a thread.
- Thread: heavyweight compared to a coroutine
- Coroutine: lightweight compared to a thread.
A coroutine is defined as a function.
A thread is an object created and managed by the underlying operating system and represented in Python as a threading.Thread object.
- Thread: Managed by the operating system, represented by a Python object.
This means that coroutines are typically faster to create and start executing and take up less memory. Conversely, threads are slower than coroutines to create and start and take up more memory.
The cost of starting a coroutine is a function call. Once a coroutine is active, it uses less than 1 KB of memory until it’s exhausted.
— PAGE 267, EFFECTIVE PYTHON, 2019.
Coroutines execute within one thread, therefore a single thread may execute many coroutines.
Many separate async functions advanced in lockstep all seem to run simultaneously, mimicking the concurrent behavior of Python threads. However, coroutines do this without the memory overhead, startup and context switching costs, or complex locking and synchronization code that’s required for threads.
— PAGE 267, EFFECTIVE PYTHON, 2019.
You can learn more about threads in the guide:
4.6 协程与进程¶
4.6 Coroutine vs Process
协程比进程更轻量。
事实上，线程比进程更轻量。
进程是一个计算机程序。 它可能有一个或多个线程。
Python 进程实际上是 Python 解释器的一个单独实例。
进程与线程一样，由底层操作系统创建和管理，并由 multiprocessing.Process 对象表示。
- Process: 由操作系统管理，由Python对象表示。
这意味着协程的创建和启动速度明显快于进程，并且占用的内存也少得多。
协程只是一种特殊函数，而进程是至少具有一个线程的解释器的实例。
您可以在指南中了解有关 Python 进程的更多信息：
A coroutine is more lightweight than a process.
In fact, a thread is more lightweight than a process.
A process is a computer program. It may have one or many threads.
A Python process is in fact a separate instance of the Python interpreter.
Processes, like threads, are created and managed by the underlying operating system and are represented by a multiprocessing.Process object.
- Process: Managed by the operating system, represented by a Python object.
This means that coroutines are significantly faster than a process to create and start and take up much less memory.
A coroutine is just a special function, whereas a Process is an instance of the interpreter that has at least one thread.
You can learn more about Python processes in the guide:
4.7 Python 何时添加了协程¶
4.7 When Were Coroutines Added to Python
协程扩展了 Python 中的生成器。
长期以来，生成器一直在慢慢地朝着成为一流协程的方向发展。
我们可以探索 Python 的一些主要变化来添加协程，我们可以将其视为 asyncio 概率添加的子集。
像 send() 和 close() 这样的新方法被添加到生成器对象中，以允许它们更像协程。
这些是在 Python 2.5 中添加的，并在 PEP 342 中进行了描述.
此 PEP 对生成器的 API 和语法提出了一些增强，使它们可以用作简单的协程。
随后，允许生成器发出PEP 334中描述的暂停异常和停止异常.
该 PEP 提出了一种基于迭代器协议扩展的有限协程方法。 目前，迭代器可能会引发 StopIteration 异常来指示它已完成生成值。 该提案为此协议添加了另一个例外，SuspendIteration，它表明给定的迭代器可能有更多的值要生成，但目前无法这样做。
通过 asyncio 模块在 Python 中使用现代协程的绝大多数功能在 PEP 3156 中进行了描述，并在 Python 3.3 中添加。
这是从 Python 3.3 开始的 Python 3 中异步 I/O 的提案。 考虑一下 PEP 3153 中缺少的具体提案。该提案包括可插入事件循环、类似于 Twisted 中的传输和协议抽象，以及基于 (PEP 380) 产量的更高级别调度程序。 建议的包名称是 asyncio。
— PEP 3156 – ASYNCHRONOUS IO SUPPORT REBOOTED: THE “ASYNCIO” MODULE
基于生成器的第二种协程方法已添加到 Python 3.4 作为 Python 生成器的扩展。
协程被定义为使用 @asyncio.coroutine 装饰器的函数。
协程是通过 asyncio 模块使用 asyncio 事件循环执行的。
协程可以通过 “yield from” 表达式挂起并执行另一个协程
例如：
# 在 Python 3.4 中定义自定义协程
@asyncio.coroutine
def custom_coro():
# 挂起并执行另一个协程
yield from asyncio.sleep(1)
“yield from” 表达式在 PEP 380 中定义。
为生成器提出了一种语法，将其部分操作委托给另一个生成器。 这允许将包含“yield”的代码部分分解出来并放置在另一个生成器中。
“yield from” 表达式仍然可在生成器中使用，尽管它是一种已弃用的在协程中暂停执行的方法，有利于 “await” 表达式。
注意：对基于生成器的协程的支持已被弃用并在 Python 3.11 中删除。 基于生成器的协程早于 async/await 语法。 它们是Python生成器，使用yield from表达式来等待Futures和其他协程。
我们可以说协程是在 3.5 版本中作为一流对象添加到 Python 中的。
这包括对 Python 语言的更改，例如 “async def”、“await”、“async with” 和 “async for” 表达式 作为协程类型。
这些更改在 PEP 492 中进行了描述。
建议使协程成为Python中一个适当的独立概念，并引入新的支持语法。 最终目标是帮助在 Python 中建立一个通用的、易于理解的异步编程思维模型，并使其尽可能接近同步编程。
现在我们知道了什么是协程，让我们仔细看看如何在 Python 中使用它们。
被 python 并发 API 淹没了吗？
寻求解脱，下载我的免费 Python 并发思维导图
Coroutines extend generators in Python.
Generators have slowly been migrating towards becoming first-class coroutines for a long time.
We can explore some of the major changes to Python to add coroutines, which we might consider a subset of the probability addition of asyncio.
New methods like send() and close() were added to generator objects to allow them to act more like coroutines.
These were added in Python 2.5 and described in PEP 342.
This PEP proposes some enhancements to the API and syntax of generators, to make them usable as simple coroutines.
Later, allowing generators to emit a suspension exception as well as a stop exception described in PEP 334.
This PEP proposes a limited approach to coroutines based on an extension to the iterator protocol. Currently, an iterator may raise a StopIteration exception to indicate that it is done producing values. This proposal adds another exception to this protocol, SuspendIteration, which indicates that the given iterator may have more values to produce, but is unable to do so at this time.
The vast majority of the capabilities for working with modern coroutines in Python via the asyncio module were described in PEP 3156, added in Python 3.3.
This is a proposal for asynchronous I/O in Python 3, starting at Python 3.3. Consider this the concrete proposal that is missing from PEP 3153. The proposal includes a pluggable event loop, transport and protocol abstractions similar to those in Twisted, and a higher-level scheduler based on yield from (PEP 380). The proposed package name is asyncio.
— PEP 3156 – ASYNCHRONOUS IO SUPPORT REBOOTED: THE “ASYNCIO” MODULE
A second approach to coroutines, based on generators, was added to Python 3.4 as an extension to Python generators.
A coroutine was defined as a function that used the @asyncio.coroutine decorator.
Coroutines were executed using an asyncio event loop, via the asyncio module.
A coroutine could suspend and execute another coroutine via the “yield from” expression
For example:
# define a custom coroutine in Python 3.4
@asyncio.coroutine
def custom_coro():
# suspend and execute another coroutine
yield from asyncio.sleep(1)
The “yield from” expression was defined in PEP 380.
A syntax is proposed for a generator to delegate part of its operations to another generator. This allows a section of code containing ‘yield’ to be factored out and placed in another generator.
The “yield from” expression is still available for use in generators, although is a deprecated approach to suspending execution in coroutines, in favor of the “await” expression.
Note: Support for generator-based coroutines is deprecated and is removed in Python 3.11. Generator-based coroutines predate async/await syntax. They are Python generators that use yield from expressions to await on Futures and other coroutines.
We might say that coroutines were added as first-class objects to Python in version 3.5.
This included changes to the Python language, such as the “async def“, “await“, “async with“, and “async for” expressions, as well as a coroutine type.
These changes were described in PEP 492.
It is proposed to make coroutines a proper standalone concept in Python, and introduce new supporting syntax. The ultimate goal is to help establish a common, easily approachable, mental model of asynchronous programming in Python and make it as close to synchronous programming as possible.
Now that we know what a coroutine is, let’s take a closer look at how to use them in Python.
Overwheled by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
创建日期: 2024年9月4日