Coder Legion - Recent posts tagged asynchronous-programming

Async/Await in Production: The Bug That Only Appeared Under Load

Tue, 28 Apr 2026 16:22:39 +0000

Async/Await in Production: The Bug That Only Showed Up Under Load

Most people learn async/await in a clean little test project.

await GetDataAsync();

Looks simple. Feels safe.

Then you push to production, traffic hits, and async/await quietly wrecks your backend. This is a real failure pattern: thread starvation mixed with blocking async calls causing cascading latency spikes.

It Worked Fine Locally

Typical .NET backend API:

ASP.NET Core
SQL Server
External HTTP calls
Async/await everywhere (or so we thought)

Code looked like this:

[HttpGet]
public IActionResult GetUser(int id)
{
    var user = _userService.GetUserAsync(id).Result;
    return Ok(user);
}

Nothing obviously wrong if you come from a sync background.

Locally? Fast. Stable. No issues.

First Symptom: Random Slow Requests

In production under moderate traffic:

Some requests take 200ms
Others randomly spike to 5–10 seconds
CPU is fine
DB is fine

Nothing obvious blocking. That's the problem.

The Real Issue

This line is the killer:

_userService.GetUserAsync(id).Result;

Or .GetAwaiter().GetResult().

What happens:

Async method starts
Hits an await
Thread frees up
You immediately block waiting for the result

Now combine with ASP.NET thread pool behavior.

Thread Pool Starvation

Under load:

Thread pool has limited threads
Each request blocks a thread waiting on async work
But that async work needs threads to resume

So threads wait for threads that are already blocked. Not a classic deadlock, but the system acts like one. Latency explodes.

Why Local Didn't Catch It

Locally: low concurrency, plenty of threads, no pressure.

Production: many concurrent requests, real API latency, real DB spikes. Everything amplifies the problem.

The Fix

[HttpGet]
public async Task<IActionResult> GetUser(int id)
{
    var user = await _userService.GetUserAsync(id);
    return Ok(user);
}

And inside services:

public async Task<User> GetUserAsync(int id)
{
    return await _repository.GetUserAsync(id);
}

Async all the way up. No exceptions.

Hidden Trap

Sync over async in libraries:

Task.Run(() => _service.CallAsync()).Result

Or legacy code forcing sync wrappers. Spreads the problem silently across layers.

Another Silent Killer

Context capture:

await SomethingAsync(); // captures context

Fix for backend services:

await SomethingAsync().ConfigureAwait(false);

Avoids unnecessary context switching overhead.

How We Found It

We eventually saw:

Thread pool queue length spikes
High blocked thread count
Requests stuck in "waiting" state

After removing all .Result usage:

Latency dropped
Throughput roughly doubled
CPU normalized

Bottom line: Never mix .Result or .Wait() with async code. Async has to be end to end. Thread pool starvation is silent but deadly. Production is not local.

Asynchronous Python: A Beginner’s Guide to asyncio

Tue, 14 Oct 2025 05:51:37 +0000

Howdy Folks: Ever wondered about asynchronous processes in programming? Join us on a journey into the realm of asynchronous Python with a focus on the asyncio module. We’ll demystify the concepts, delve into syntax, and showcase how asyncio empowers concurrent task execution. Get ready to harness the power of asynchronous Python and elevate your programming prowess!

What is an Asynchronous process anyways?
In the world of programming, there are tasks that take varying amounts of time to complete. Some tasks may be quick, while others might require waiting for external resources, such as data from a server or user input. As developers, we often encounter situations where we need to perform multiple tasks concurrently without waiting for each one to finish before starting the next. This is where asynchronous programming comes into play, and the asyncio module in Python is a powerful tool to achieve this.

Importing the asyncio Module

Ok folks, the first step in utilizing the asyncio module is to import it into our Python script. This can be done using the following line of code:

import asyncio

This line makes all the asynchronous magic provided by the asyncio module available for use in our script. easy peasy, lemon squeasy, right?

Understanding Asynchronous Tasks

In traditional synchronous programming, tasks are executed one after the other, and the program waits for each task to complete before moving on to the next. In asynchronous programming, we can mark certain functions as asynchronous using the async keyword, allowing them to run concurrently with other tasks.

Let’s take a simple example to illustrate this concept, folks:

async def task1():
    print("Wake Up started")
    await asyncio.sleep(1)
    print("Wake Up completed")

Oh, what is that? Ok, allow me to explain it: In this code, the async def declaration indicates that the function task1 is asynchronous. The await asyncio.sleep(1) line simulates a task that takes 1 second to complete. While waiting for this task, other asynchronous functions can execute.

The Main Function and asyncio.gather

Now, in order to coordinate and execute multiple asynchronous tasks, we use the asyncio.gather method within a main asynchronous function.

Let’s take a look at this example, champs:

async def main():
    await asyncio.gather(task1(), task2(), task3())

In this code, the main function is marked as asynchronous using async def. The asyncio.gather method is then used to execute the tasks task1, task2, and task3 concurrently. The await keyword is used before asyncio.gather to ensure that the program waits for the completion of all tasks.

Bringing It All Together

Now, let’s put everything together into a complete script:

import asyncio

async def task1():
    print("Wake Up started")
    await asyncio.sleep(1)
    print("Wake Up completed")

async def task2():
    print("Take a shower started")
    await asyncio.sleep(2)
    print("Take a shower completed")

async def task3():
    print("Have Breakfast started")
    await asyncio.sleep(4)
    print("Have Breakfast completed")

async def main():
    await asyncio.gather(task1(), task2(), task3())

asyncio.run(main())

When you run this script, you’ll see the output reflecting the asynchronous execution of tasks. Each task begins, and while waiting for the time-delayed sleep to complete, other tasks proceed concurrently.

Wrapping Up

In this article, we explored the basics of asynchronous programming in Python using the asyncio module. We learned how to mark functions as asynchronous, use the await keyword to wait for asynchronous tasks, and coordinate multiple tasks using asyncio.gather. Asynchronous programming enables more efficient use of resources and improves the responsiveness of applications, making it a valuable skill for developers working on various projects.

I trust you found this article both educational and enjoyable. I’ve strived to simplify the content for a quick and effective learning experience. Happy coding, and may your Python endeavors always thrive in your daily life!

Enjoyed the article? Your feedback fuels more content.
Share your thoughts in a comment.
No time to read now? Well, Bookmark for later.
If it helped, click on share & pass it on, dude!

The Quest Continues: Porting the Word Game With AsyncSSH

Sun, 04 May 2025 23:33:09 +0000

Last week, we reimplemented our 20-questions variant, and it received quite a bit of attention. So far we built a web experience, as well as a command line interface for the game. However, unlike the web version, the command line interface is strictly local. There’s no way to share the game with a fellow friend unless we walk them through the setup. Coincidentally, a former coworker recently told me about hosting applications over secure shell protocol in Golang. That makes me wonder, how easily can we replicate it in Python?

A cute illustration from Copilot on the topic.

The Spark of an idea

The secure shell protocol (SSH) is usually what we use to log into another machine for work. Yet, there are a lot of interesting applications hosted over SSH. I remember there was one that would play back a Star Wars movie in ASCII art upon connecting successfully. Late last year, my coworker told me about a coffee shop selling freshly brewed coffee over SSH. We both found that idea intriguing.

He suspected the application was made with wish, a Golang library. That prompted me to see if there is an equivalent for Python, but I only found Paramiko at the time. After a quick glance at the documentation, I decided it was too much work and moved on. It does seem like it is capable of achieving the goal — serving a CLI application over SSH, though.

Yes, the em-dash is intentional.

Photo by Vitaly Gariev on Unsplash

Keeping in touch as usual, and he would bring up the wish project from time-to-time. Then fast-forward to last week, after I was done with the reimplementation of our little word game, I stumbled upon another library called AsyncSSH. Like Paramiko, it can be used to create an SSH server as well. The relatively more extensive examples also seemed helpful.

After some exchanges with Gemini, I received a generated proof-of-concept code that looked workable. After completing and publishing the article on the game, I finally had time to properly explore the topic. Read on to join the ride together in building an interactive application served over SSH.

Finding the Right Tool: Paramiko vs AsyncSSH

Like we briefly mentioned previously, Paramiko and AsyncSSH are libraries we can use to build SSH Server applications (they are also excellent as SSH client libraries). Though the former garnered more stars on GitHub, the amount of information on how to build an SSH Server application is surprisingly sparse. From what I read, it also required quite extensive setup.

The discovery of AsyncSSH was completely unintentional. It was buried deep in the search result when I was looking for a cohesive tutorial on Paramiko. While the design of the documentation site looked as if it was built in Python 2.0 era, the extensive collection of examples is extremely helpful. I particularly liked the simplicity offered by AsyncSSH in comparison.

Photo by Agence Olloweb on Unsplash

Our goal for this week is a fairly simple application sporting a simple REPL-like experience. The key of this exercise is to learn, hence efficiency isn’t a deciding factor. Therefore, the simplicity and developer friendly documentation make the library a sensible choice for our project this week.

Two Paths and a Relearning

Photo by Ross Sneddon on Unsplash

The quick proof-of-concept code I came out with the help of Gemini, by referring to the documentation, consists of 3 components. Firstly, we have the actual SSH Server class, subclassing asyncssh.SSHServer.

import asyncssh

class MySSHServer(asyncssh.SSHServer):
    def begin_auth(self, username):
        logger.info("logging in user:%s", username)
        return False

    def session_requested(self):
        return MyREPLSession()

This class is where we define all the authentication rules. For the purpose of this article, we are skipping all of them to maintain simplicity. Thus, we are returning False in begin_auth to indicate no authentication is needed. Do refer to the documentation for information on the topic before publishing applications in production.

The MyREPLSession is a class inheriting asyncssh.SSHServerSession, which defines the actual application logic we intend to serve over the server. The skeleton is shown as below:

class MyREPLSession(asyncssh.SSHServerSession):
    _chan = None
    _buffer = ""

    def connection_made(self, chan: asyncssh.SSHLineEditorChannel) -> None:
        self._chan = chan

    def pty_requested(
        self,
        term_type: str,
        term_size: tuple[int, int, int, int],
        term_modes: dict[int, int],
    ) -> bool:
        logger.info("pty received")
        return True

    def shell_requested(self) -> bool:
        logger.info("shell received")
        return True

    def data_received(self, data: str, datatype: int | None) -> None:
        # TODO The actual application
        pass

    def eof_received(self):
        assert isinstance(self._chan, asyncssh.SSHLineEditorChannel)

        self._chan.exit(0)
        return False

    def signal_received(self, signal):
        assert isinstance(self._chan, asyncssh.SSHLineEditorChannel)

        self._chan.exit(0)

    def break_received(self, signal):
        assert isinstance(self._chan, asyncssh.SSHLineEditorChannel)

        self._chan.exit(0)

There are a couple of key methods we need to implement. The first is the connection_made, in our setup we are expecting a asyncssh.SSHLineEditorChannel object. All the input and output through SSH would be handled by the channel object, hence it is crucial to keep a reference of it. Methods ending with the _received suffix are event handlers, and in our toy project, we simply close the channel when no more input is sent (upon EOF or on terminate/kill signals).

We will discuss data_received event handler later.

Lastly, we need to tie everything together through asyncssh.create_server. As implied by the library name, the function returns a coroutine object, and hence we would need to properly set up through AsyncIO. Feel free to revisit our previous article on how to write an asynchronous program. But for this example, we are adapting the example shown in the documentation.

AsyncIO Task Management: A Hands-On Scheduler Project

import asyncio

async def main() -> None:
    try:
        # Ensure ssh-host key exists before running
        await asyncssh.create_server(
            MySSHServer,
            "0.0.0.0",
            8022,
            server_host_keys=["./ssh-host"],
            line_editor=True
        )
        print("SSH server listening on port 8022...")
    except Exception as exc:
        print(f"Error starting server: {exc}")

if __name__ == " __main__":
    loop = asyncio.new_event_loop()

    loop.run_until_complete(main())
    loop.run_forever()

We previously used asyncio.run to run the main coroutine, and set up a ShutdownHandler working with an exit event object to end a program. On the other hand, this approach was likely written before asyncio.run was introduced, with the manual management of event loop being the telling sign. The run_forever() here also eliminates the hack where people put an infinite loop after asyncssh.create_server.

Essentially, the code snippet creates the server, where the server behaviour is defined by the MySSHServer, and we set line_editor=True to be explicit (as we are expecting user interaction in our session). Remember to create an SSH key for the application.

$ ssh-keygen -q -N "" -t ed25519 -f ./ssh-host

As mentioned earlier, we are building a very simple REPL-like application. Just as a refresher, REPL stands for read-evaluate-print-loop, which aptly fits our case here. We read user input, get DSPy to process it with the LLM in background, print the generated response, rinse-and-repeat. The skeleton of the flow is shown as below:

def main() -> None:
    game_setup_and_welcome_speech()

    while True:
        match input("> "):
            case command if command == "quit":
                break
            case attempt:
                process_and_print_reply(attempt)

Now we can populate our data_received handler. This method is called whenever our user is sending input through the SSH session. Unfortunately, we cannot assume the data is always ending with a newline (user pressing enter). Sometimes, the handler may have accumulated a couple of questions and send them in one go. The function may also get called while the user hasn’t done typing. For this, we would have to handle both cases, as shown below:

class MyREPLSession(asyncssh.SSHServerSession):
    _buffer = ""

    def data_received(self, data: str, datatype: int | None) -> None:
        assert isinstance(self._chan, asyncssh.SSHLineEditorChannel)

        self._chan.write(f"Welcome to my SSH server, {username}!\n\n")
        self._buffer += data

        lines = self._buffer.split("\n")
        for line in lines[:-1]:
            match line:
                case command if command == "quit":
                    self._chan.exit(0)
                    break

                case attempt if attempt:
                    process_and_print_reply(attempt)

        self._buffer = lines[-1]

Remember we talked about handling user input and output through the SSHLineEditingChannel object we received in connection_made? Output of our application is done through its write() method, just like how we output to stdout (to be displayed on the terminal emulator) through print() usually.

Let’s test the application.

A seemingly stalled game session

While the application worked, the experience wasn’t ideal. My modest GPU took quite a bit of time to generate a response for every user input, and somehow the cursor stalled and wouldn’t move to another line whenever the user pressed enter. It somehow gave an impression of the application just died after every enter.

Photo by Josh Redd on Unsplash

I was ready to move on, and start preparing to write, until I came across another example that fits the use-case. Now we have a code that looked a lot like the original skeleton.

async def handle_client(process: asyncssh.SSHServerProcess) -> None:
    username = process.get_extra_info("username")
    process.stdout.write(f"Welcome to my SSH server, {username}!\n\n")

    game_setup_and_welcome_speech()

    while True:
        match await process.stdin.readline():
            case command if command == "quit":
                break

            case attempt:
                process_and_print_reply(attempt)

    process.exit(0)

Previously we returned output through the SSHLineEditingChannel object, here we are doing it through the SSHServerProcess object. On the other hand, instead of input(), we read it through the readline() method in the stdin (an SSHReader object) property.

Then, we need to remove the session_requested method from MySSHServer, and define the process_factory argument in asyncssh.create_server to point to our new coroutine.

await asyncssh.create_server(
    MySSHServer,
    "0.0.0.0",
    8022,
    server_host_keys=["./ssh-host"],
    line_editor=True,
    process_factory=handle_client,
)

Re-run the application again, and to my surprise, it worked almost exactly like the original CLI version. For our project, the second approach definitely worked better. Nonetheless, the first attempt was also crucial, as it gives an idea on how input is being sent in a SSH session.

The choice of naming the property holding a SSHReader object as stdin, and the fact it reads input through a method named readline piqued my curiosity. I knew we could reach sys.stdin to read content redirected from the previous command (via the pipe | operator), or file content via the < operator (in Unix-like environments). That said, I didn’t know reading user input interactively in a running application is done through stdin as well.

My first experience writing a production CLI application started with PHP (very strange, yes). That was my first time learning about the relationship between the operators and stdin, stdout, and stderr through my supervisor. The scripts didn’t really need to collect input interactively, and given the similarity of PHP and C, I couldn’t find a scanf equivalent so I assumed PHP doesn’t have such feature.

I was wrong.

Photo by Mr. Bochelly on Unsplash

After some exchanges with Gemini, I realized I could use readline() in PHP for the task. Which itself is similar to scanf and input in Python. Alternatively, I could also use fgets(STDIN) to do the job, just that I need to figure out a way to print a prompt prior to the input collection. On the other hand, if I redirect some content into a Python application, if I call input(), it would read the content in stdin, until it encounters a newline character. Essentially, it roughly equates

import sys

sys.stdin.readline().strip()

Back to the topic, the toy example skipped many other aspects besides the authentication work we mentioned earlier. Unfortunately, due to the sheer scope and complexity, it is almost impossible for the developer to cover all use cases in the examples. Fortunately, the API documentation is very well-written, and I assume it could be useful for people who know more about the protocol.

We are lucky to live in an era where a LLM chatbot like Gemini is readily available, to provide help in navigating the documentation. Besides that, it also serves as a valuable companion to help gathering complementary information, like how I relearned stdin. Even so, critical thinking is still crucial while evaluating the response to avoid falling victim to AI hallucination.

Bridging Game and Server

Photo by Phil Hearing on Unsplash

We finally assembled all the pieces together and ported the game to be served over SSH. The concrete implementation can be found at the GitHub repository for those who are interested. Both the CLI version and SSH version (the handle_client variant) are very similar, as shown below

async def handle_client(process: asyncssh.SSHServerProcess) -> None:
    username = process.get_extra_info("username")
    process.stdout.write(f"Welcome to my SSH server, {username}!\n\n") # vs print()

    game_setup_and_welcome_speech()
    while True:
        match await process.stdin.readline(): # vs input()
            case command if command == "quit":
                break
            case command if command == "end" and answer is not None:
                answer = None
            case attempt:
                process_and_print_reply(attempt)

    process.exit(0)

The major difference between the two, is the use of process.stdout.write() to replace print(), and process.stdin.readline() to replace input (though a new coroutine called prompt_input is written to compose a process.stdout.write() to display prompt text).

async def prompt_input(process: asyncssh.SSHServerProcess, prompt: str) -> str:
    process.stdout.write(prompt)
    return await process.stdin.readline()

The REPL-like experience makes the word game a very synchronous experience. A response is only generated upon user submission of input. Therefore the await is necessary while expecting user input is needed to ensure order. Other than that, the asynchronous nature of AsyncSSH does not affect the game’s logic much. As each game session is independent, each time a new user connects, the library will run handle_client separately for each of them.

Once the log message shows “SSH server is listening …” after executing ssh.py, it is ready to be connected to with the following command.

$ ssh -p 8022 localhost

Considering we didn’t define any authentication methods in MySSHServer (which is definitely a bad practice), it should display a welcome message and the game is started right away.

In this article, we explored two different strategies to host our little word game over SSH. The two approaches have their own pros and cons, but it does show how AsyncSSH makes the development experience quite intuitive. Even though I do not have much experience doing this in Golang with wish (the documentation situation is also quite bad there), I enjoy the time spent doing this with AsyncSSH.

The Game Goes Live

Photo by Alessandro Erbetta on Unsplash

The point of the exercise is mainly on learning and discovery. If our game were developed with a text-based UI through Textual, or even Typer, some extra work is needed to ensure I/O is redirected properly.

The experience of porting the library to SSH is a series of happy coincidences. It started with a spark from a discussion with my former coworker, and it led to a series of discoveries. This article is a culmination of the whole journey, sharing what I learned throughout. It turns out building an application to be served over SSH just needed a change in how I/O is done. Gemini also helped immensely throughout, deepening my understanding.

We went through the whole process by configuring a server by subclassing a asyncssh.SSHServer, exploring two different ways to serve applications through process_factory and inheriting asyncssh.SSHServerSession, then getting it running via asyncssh.create_server.

Hopefully this article sparks an interest and we see more interesting applications hosted over SSH.

This article was written with assistance from Gemini, a large language model by Google. Gemini provided editorial help, fixing language errors and checking for article flow. However, the code and voice are entirely my own. If you’d like to collaborate or discuss job opportunities, please connect with me on LinkedIn or via Medium.

AsyncIO Task Management: A Hands-On Scheduler Project

Tue, 01 Apr 2025 06:46:58 +0000

We discussed Awaitables last week. The article covered coroutines, tasks and futures as well as a quick introduction to the event loop. Let’s build an example task management project to continue our learning journey.

A cute cartoon generated by Microsoft Copilot on the topic

Introduction: Building Our Task Scheduler

In the previous article, we mentioned briefly on event loops. Think of it as a timetable-like structure, which is executed throughout the lifetime of an application. Awaitables are then scheduled for concurrent execution within it. In recent releases, the Python developers provided more abstractions and reduced the need to manually manage event loops.

In the article on programming a chatbot, we created a new process each for the chatbot and the web application. We then call asyncio.run in each of the processes to execute the first coroutine. So let’s copy that to bootstrap our sample application:

import asyncio

async def main():
    await asyncio.sleep(5)

if __name__ == ' __main__':
    asyncio.run(main())

If we run the program, it will run for 5 seconds through asyncio.sleep, and then exits. As mentioned last week, calling a coroutine (asyncio.sleep) would not cause it to execute. Omitting the await keyword will result in an error. This is because the coroutine object returned by asyncio.sleep(5) does not get scheduled for execution.

In the snippet, calling asyncio.run creates an event loop, schedule the coroutine (main()) for execution, and then perform clean-up after completion.

We are building a task management system, where we submit tasks, and then run a check for result manually. So the first thing we need is a simple user interface. I like the simplicity of repl-like (read-evaluate-parse-loop) interface, so we build one. To interact with our application, users can issue simple commands. We start with 2 commands below, by adapting from the implementation we did last week.

dex : fetch the name of a Pokémon from the Pokédex based on the number
fib : Returns the th Fibonacci number

Let’s start by implementing dex and fib, each represents an I/O bound and CPU-bound operation respectively.

async def dex(client: httpx.AsyncClient, id: int) -> str:
    assert isinstance(id, int)

    response = await client.get(f"https://pokeapi.co/api/v2/pokemon/{id}/")

    return f"The pokemon with id {id} is {response.json()['name']}"

def fib(nth: int) -> str:
    assert isinstance(nth, int) and nth > 0

    result = ()

    for i in range(1, nth + 1):
        match i:
            case 1:
                result = (0,)

            case 2:
                result += (1,)

            case _:
                result = result[1:] + (sum(result),)

    assert len(result) > 0

    return f"The {nth}th fibonacci number is {result[-1]}"

For dex, we want to reuse the httpx.AsyncClient, so we adapt our code to require it in the arguments.

Tasks and Futures in Action

Assembling the repl-like interface would need a little bit of work. We need to be able to parse the command entered by our users and schedule the correct task. For this, we use funcparserlib as featured in a previous Advent of Code article. We are skipping the implementation detail for parser and lexer in this article, and only focus on the evaluation part, as shown below:

async def evaluate(
    client: httpx.AsyncClient,
    expression: Expression,
) -> Awaitable[str] | str:
match expression:
    case Expression(command=Command.DEX):
        return asyncio.create_task(dex(client, expression.args[0]))

    case Expression(command=Command.FIB):
        return asyncio.create_task(asyncio.to_thread(fib, expression.args[0]))

    case _:
        raise Exception("Unknown command")

We want to enable users to check the progress of tasks later, so we return them as tasks. For dex requests, it is implemented as a coroutine, hence we just wrap it as a task, and return it. However, for fib, as it is CPU-bound, it will not benefit much from asynchronous execution. As discussed last week, we need to turn it into a future, so it can be executed as a thread of process, as shown below

loop = asyncio.get_running_loop()

# get the 100th Fibonacci number
future = loop.run_in_executor(None, fib, 100)

# later in the execution
result = await future

As part of the abstraction effort, asyncio.get_running_loop() will fetch the event loop created by asyncio.run we called earlier. However, if the code snippet is executed after the main() coroutine is completed, it will result in an error. The first argument in run_in_executor expects either a ProcessPoolExecutor, or a ThreadPoolExecutor, but will use a default ThreadPoolExecutor if None is passed. For most cases, it is sufficient to just use the default, and hence we can turn it into a task instead

fib_coro = asyncio.to_thread(fib, 100)

task = asyncio.create_task(fib_coro)

# later
result = await task

Now we have the executor done, let’s assemble the repl. We use prompt-toolkit to prompt the user for command, and then it does the following operations in sequence

Read the user entered command and arguments
Parse the command and arguments, and return an Expression object
Evaluate the expression object, and schedule tasks with asyncio.create_task
Loop back to step 1

The main coroutine, is now written as follows, with the corresponding steps are annotated as comments in the snippet

async def main() -> None:
    t, g = TOKENIZER, grammar()
    session = PromptSession()

    tasks = []

    async with httpx.AsyncClient() as client:
        # step 4: loop
        while True:
            # step 1: read input
            line = await session.prompt_async("> ", handle_sigint=False)

            match result := await thread_last(
                line,
                # step 2a: tokenize the input
                (tokenize, t),
                # step 2b: parse the input
                (parse, g),
                # step 3: evaluate
                (evaluate, client),
            ):
                case asyncio.Task():
                    tasks.append((line, result))
                    print(f"Task {len(tasks)} is submitted")

                case _:
                    print(result)

Some notes on the snippet:

t is for the definition of tokens (e.g, a sequence of letters is the name of a command)
g is for the definition of the commands (what dex and fib means)
The thread_last call in the snippet is taken from toolz. What it does is simply evaluate(client, parse(g, tokenize(t, line))) but we keep it that way to aid readability.
We collect the tasks after evaluations into a simple list, we will work with it later
In the match statement, we print task ID if the evaluator returns a Task, or the returned string otherwise.

So let’s recap what we have built so far. We should be able to run the program, and then issue commands to fetch Pokémon names, and calculate Fibonacci numbers. You may have noticed, we have yet to print the result on screen.

On the other hand, we also know that main() is calling the evaluator to schedule tasks to the event loop with asyncio.create_task(). We also see await is used when the order of execution is crucial:

When we perform a GET HTTP request, so we can parse the response JSON after receiving it
When we are receiving a user input, as we need to pass the input to parser later
We need to wait for evaluator to create a task, before we loop back to receiving another input from the user

So what are the differences between the two, when should we use asyncio.create_task() and await? As a rule of thumb, if the order of execution does not matter, or if we only care about the outcome much later, use asyncio.create_task. If the coroutine does something that is needed later as shown above, then use await to ensure it is run as soon as possible, before we move on to the next line in code.

Managing Task Execution and Displaying Results

Now we can start working on the tracker, and then build a dashboard to list tasks. These are the two command we want to build next:

job : We show the result of computation, or a message indicating it is still waiting for a result
dash: We draw a simple table to show some information about all submitted tasks

Starting with the actual function again, first we have one for job

def job(task: tuple[str, asyncio.Task[str]]) -> str:
    try:
        return task[-1].result()

    except asyncio.InvalidStateError:
        return "Still waiting"

Each entry in the task list, is a tuple of user input, and the actual Task object. So in this function, we are interested in finding the result. However, if the task is still not ready, then calling .result() will raise InvalidStateError, then we can return a message to tell users that it is still waiting.

Then a dashboard to display an overview of the tasks

def dash(tasks: Sequence[tuple[str, asyncio.Task[str]]]) -> str:
    result = [
        "\t".join(("id", f"{'command':16s}", "done?", "result")),
        "\t".join(("==", f"{'=' * 16}", "=====", "======")),
    ]

    for num, (line, task) in enumerate(tasks, 1):
        result.append(
            "\t".join(
                (
                    str(num),
                    f"{line:16s}",
                    str(task.done()),
                    task.done() and str(task.result())[:16] or "",
                )
            )
        )

    return "\n".join(result)

The function simply returns a table-like report to show if tasks are done, and the corresponding result whenever applicable.

Then we need to expand the evaluator, to call the corresponding job tracking functions we just added above.

async def evaluate(
    client: httpx.AsyncClient,
    tasks: Sequence[tuple[str, asyncio.Task[str]]],
    expression: Expression,
) -> Awaitable[str] | str:
    match expression:
        ...

        case Expression(command=Command.JOB):
            return job(tasks[expression.args[0] - 1])

        case Expression(command=Command.DASH):
            return dash(tasks)

Previously, every time a user submits a task with either dex or fib, we return a job number. Now they can check the progress with the job command with corresponding job number, or display an overview of submitted jobs with dash.

Task Cancellation and Graceful Shutdown

We now already have a mostly complete program. While prompt-toolkit provides some handling for some signals, we can opt to do it on our own too (hence the handle_sigint=False in our session.prompt_async). For that, we start by declaring the ShutdownHandler, to gracefully shut down all scheduled unfinished tasks:

@dataclass
class ShutdownHandler:
    signal: int | None = None

    async def __call__ (self) -> None:
        logger.info("Shutting down tasks. signal:%s", self.signal or "")

        tasks = tuple(
            task for task in asyncio.all_tasks() if task is not asyncio.current_task()
        )

        for task in tasks:
            task.cancel()

        await asyncio.gather(*tasks, return_exceptions=True)

        asyncio.get_running_loop().stop()

We are applying the __call__ dunder discussed previously to avoid usage of functools.partial. In the coroutine, we are shutting down all the remaining tasks, and then stop the event loop (hence ending the asyncio.run we did in the beginning). Notice we are excluding the current coroutine so asyncio.get_running_loop().stop() will get to run.

Next, we set up the shutdown handler to run, whenever the program receives certain signals from the operating system. In our example, we are handling SIGTERM, SIGHUP and SIGINT. We can then register the ShutdownHandler we just implemented to the event loop to respond to the signals we listed.

Now we can proceed to add 2 more commands, namely

kill : Kill the job number
quit: Exit the program

The corresponding functions for both commands are shown below:

def kill(task: tuple[str, asyncio.Task[str]]) -> str:
    task[-1].cancel()

    return f'Task "{task[0]}" is killed'

def quit(shutdown_handler: Callable[..., Coroutine[None, None, None]]) -> str:
    asyncio.create_task(shutdown_handler())

    return "Exiting"

Every time the method .cancel() is called on a task, a CancelledError is raised. If the task is executing SQL statements in a transaction, this serves as an opportunity to quit the transaction, and rollback to undo changes done so far. But in some cases, we can just ignore the error and quit quietly. For instance, we can just suppress the error in our main() coroutine. While we revisit main(), we need to also ensure init() is called to set up the ShutdownHandler().

from contextlib import suppress

def init() -> None:
    loop = asyncio.get_running_loop()

    for sig in (signal.SIGTERM, signal.SIGHUP, signal.SIGINT):
        handler = ShutdownHandler(sig)

        loop.add_signal_handler(sig, lambda: asyncio.create_task(handler()))

async def main() -> None:
    with suppress(asyncio.CancelledError):
        init()

        t, g = TOKENIZER, grammar()
        session = PromptSession()

        ...

And the updated evaluator function is slightly revised to add support for the two new commands:

async def evaluate(
    client: httpx.AsyncClient,
    tasks: Sequence[tuple[str, asyncio.Task[str]]],
    shutdown_handler: Callable[..., Coroutine[None, None, None]],
    expression: Expression,
) -> Awaitable[str] | str:
    match expression:
        ...

        case Expression(command=Command.KILL):
            return kill(tasks[expression.args[0]] - 1)

        case Expression(command=Command.QUIT):
            return quit(shutdown_handler)

Background Tasks and Event Management

The implementation seems straightforward, as we are constantly interacting with the main() coroutine. It is always waiting for user commands, and by exiting we just terminate the infinite while loop. However, there are cases where the application would just run autonomously. For instance we have a simple application as follows:

async def background_task():
    while True:
        await asyncio.sleep(10)

        print("Print something")

async def main():
    # send work to execute in the background
    asyncio.create_task(background_task())

    # do some work
    ultimate_answer = 42

    # hmm, we are done here

if __name__ == ' __main__':
    asyncio.run(main())

A similar pattern can be observed in the article where we discussed chatbot programming, where the background_task is a coroutine to consume messages from a queue. In this mini example program, it will end silently, and the first print() in background_task() will not happen, because main() ended so soon. In the past, I have seen people attaching another infinite loop at the end of main() so background_task() gets to run indefinitely.

async def main():
    ...

    # let's pretend we are doing work
    while True:
        pass

While it seems to work, such that the application seems to continue to run. However, background_task() is not printing because the infinite loop in main() is blocking other coroutines from running. As a workaround, we could replace pass with something like await asyncio.sleep(2), but it felt like a hack and I don’t quite like it.

Alternatively, we could create an asyncio.Event object, and then pass it to main(). Then we adapt the ShutdownHandler discussed previously, to receive an event object, as shown below:

@dataclass
class ShutdownHandler:
    exit_event: asyncio.Event
    signal: int | None = None

    async def __call__ (self) -> None:
        logger.info("Shutting down tasks. signal:%s", self.signal or "")
        self.exit_event.set()

async def main(exit_event: asyncio.Event):
    loop = asyncio.get_running_loop()

    for sig in (signal.SIGTERM, signal.SIGHUP, signal.SIGINT):
        handler = ShutdownHandler(exit_event, sig)

        loop.add_signal_handler(sig, lambda: asyncio.create_task(handler()))

    # send work to execute in the background
    task = asyncio.create_task(background_task())

    # do some work
    ultimate_answer = 42

    await exit_event.wait()
    task.cancel()

if __name__ == " __main__":
    asyncio.run(main(asyncio.Event()))

So now main() now receives an event, setup the ShutdownHandler, and then upon receiving from the event, cancel the background_task(). For completeness’ sake, we can also handle the CancelledError in background_task, as follows:

async def background_task():
    try:
        while True:
            await asyncio.sleep(10)

            print("Print something")

    except asyncio.CancelledError:
        print("Bye bye")

Handling Exceptions

Before we conclude, is on error handling. When we work with asyncio enough, we will notice if an awaitable running in background fails, it is not always obvious to us. Fortunately, like signals, we can also attach a handler if an exception is raised. Back to our example project, let’s implement an ExceptionHandler for the purpose.

@dataclass
class ExceptionHandler:
    shutdown_handler: ShutdownHandler

    def __call__ (self, loop, context) -> None:
        logger.error("Application failed. context:%s", context)

        asyncio.create_task(self.shutdown_handler())

Then, add a new line to register the handler after the signal handler registration

async def init():
    ...

    loop.set_exception_handler(ExceptionHandler(ShutdownHandler()))

While this is a more a catch-all workaround, we should still properly handle exceptions in all our coroutines. For instance, in addition to CancelledError, we should probably also handle errors corresponding to network problems in dex. However we are keeping our example concise to avoid making this article into an epic novel.

Conclusion: Mastering Asynchronous Task Orchestration

That’s all I want to cover in this article. While we spent quite some time building the application, we still barely scratched the surface of the capability of AsyncIO. We discussed about how to start executing a coroutine, and then from there how to queue more tasks and awaitables into the event loop. We also discussed about exiting the program gracefully, and how to handle errors.

I owe a lot to this series of articles by Lynn Root while I was re-learning AsyncIO when rewriting bigmeow, the chatbot project. Some part of the code in our example project today is adapted from her implementation. The complete project can be found at GitHub ,,) with some slight changes.

Thank you so much for reading this far, and I shall write again, next week.

To ensure clarity and address potential language errors, this article received editorial assistance from a large language model. Please note that all ideas and concepts presented herein are my own. The code examples provided were not generated or edited by an LLM. If you’re interested in project collaboration or job opportunities, feel free to reach out to me via a message here on Medium or through LinkedIn for further discussion. This article was originally published at Medium.

Understanding Awaitables: Coroutines, Tasks, and Futures in Python

Mon, 24 Mar 2025 18:58:34 +0000

Previously, we explored chatbot building with AsyncIO. As I am committing to publish one article a week, I am constantly on the lookout for new ideas. Then I figured it may be a good opportunity to delve deeper into asynchronous programming. The writing plan eventually expanded to a series of articles, and we are starting with the concept of Awaitables today.

An illustration from copilot on the topic

To a lot of us, our first exposure to something resembles asynchronous programming would be learning to write GUI applications. A graphical application usually responds to user events, that may happen any time while it is running. We then attach a handler function to events we are interested in, however we do not have much control when it is executed.

The JavaScript Prelude: From Callbacks to Promises

Photo by Womanizer Toys on Unsplash

I first learned programming when I was in college. The assignments were designed to train us to write code with a set of a controlled scope. I graduated in the era of web 2.0, that’s when people started to explore AJAX to add dynamic behaviour to web pages. Suddenly, despite we were plagued with inconsistencies among browser’s JavaScript implementation, webpages started to feel like a GUI application.

AJAX, which is a short for Asynchronous JavaScript and XML, opens up the possibility for us to fetch data from a web server through JavaScript. With proper use, we can get a page to fetch additional information to update the page content, responding to various events defined in the document object model, DOM. Some examples of DOM events are, clicking on elements in page, elements loaded in page, or even timed events through setInterval and setTimeout.

Despite having some experience with GUI application programming, but doing it in real enterprise setting feels very different. I was still too used to structured procedural programming, where code execution is often synchronous, in a mostly predictable order. However, when it comes to an enterprise setting, the requirements are constantly changing and time is often a luxury we cannot afford. Despite having good mentor, it still took me quite a while to have a basic grasp of asychronous programming with the delicate language construct at the time.

Let's start with event handlers. Everything we see in a web page, is represented in a Document Object Model (DOM) tree, and each element is capable of responding to events. For instance, we can click on a button, and have it pop up an alert dialog through a code snippet below.

# In HTML markup
<button id="theButton">The button</button>

# In JavaScript
const button = document.getElementById("theButton")
button.addEventListener((e) => window.alert("Button is pressed"))

Then we have callbacks, which were popularized by AJAX calls. Back then, with jQuery, we could define handlers to deal with both success or failure cases. For instance, let's say we want to fetch the HTML markup of this blog (skipping error failure callback for brevity), we do

$.ajax({
    url: "https://kitfucoda.medium.com/",
    success: (data, status, xhr) => { /* process data here */ }
})

It is worth noticing that both the event handler and the success callback we defined earlier, can only access variables that were defined before their definition. The handler and callback would have no access to new external variables. On the other hand, the code outside the callback and handler do not have access to variables within them. Essentially, we can treat the code being executed separately, completely breaking off from the normal execution flow. Therefore, we do not have a way to tell if they are executed, or when they are done (or failed).

It is also possible to combine the two together. For example, combining the two examples above, we can make the page to fetch this blog after clicking on the button.

button.addEventListener(
    (e) => $.ajax({
        url: 'https://kitfucoda.medium.com',
        success: (data, status, xhr) => { /* maybe populate this somewhere */ }
    })
)

Remember we talked about how it is not possible to know if these handlers are done executing from the outside? Yes, nesting them was the only way to ensure order. If you need to fetch my other blog https://cslai.coolsilon.com AFTER finishing fetching this blog, you can do another $.ajax call within the success handler above. The growing nested callbacks eventually led to what people call a callback hell or pyramid of doom.

I didn’t follow the development of JavaScript much, as I moved on to doing Python. However we eventually had Promise, and my first encounter with it was again through jQuery. Now, it is possible to write a sequence of AJAX calls in a chain, instead of nesting endlessly. For instance

$.ajax({url: 'https://kitfucoda.medium.com/'})
    .then((response) => {
        // do something with the response
        // Now start a new ajax
        return $.ajax({url: 'https://cslai.coolsilon.com/'})
    })
    .then((response) => {
        // Do other thing with the response
    })

No more pyramid of doom, but the callback and the code outside of it still remained separate and independent to each other.

Enter AsyncIO: Awaitables Defined

My exploration of asynchronous programming in Python, was just as chaotic. When I was working for a company that offers CDN service, I spent some time learning about gevent and greenlets. It was practically my first introduction to the asynchronous world in the language. However, I left the company soon after that and didn’t have a chance to see it in action. Years later, I hacked a chatbot that barely worked, and also did a quick exercise building an ASGI websocket chatroom, without use of any frameworks. While I managed to do it, but it wasn’t a useful introduction to AsyncIO.

This article, like the one I did on chatbot, is an article I wished I read back then.

Coroutines: The Foundation of Awaitables

Photo by Kelly Sikkema on Unsplash

As a prologue to the article I am writing next week, we cover Awaitables this week in the context of AsyncIO. Understanding what they are is core to understanding how to program with AsyncIO. However, before we start, let’s talk about coroutines. A coroutine is a function defined with an async keyword. Note that if a function has any async or await statement, it has to be defined as a coroutine.

# a normal coroutine
async def a_coroutine():
    return 42

# this is invalid
def this_should_be_a_coroutine():
    await some_awaitable()
    return 42

We often write functions doing heavy IO-bound work as coroutines, such that other CPU bound computation can still take place simultaneously. To execute these coroutines, we would need an event loop. Despite the name, I prefer seeing it as a timetable like structure, a place where it accepts Awaitables (like coroutines) to be scheduled to run.

Let’s slowly build a fictional project, as a tool to explore more about asyncio in this series of articles. We start by defining a coroutine that simulates I/O operations, and use httpx to make HTTP requests. Given a national Pokédex ID, we want to return the name of the corresponding Pokémon.

async def dex(id: int) -> str:
    assert isinstance(id, int)

    async with httpx.AsyncClient() as client:
        response = await client.get(f"https://pokeapi.co/api/v2/pokemon/{id}/")

        return f"The pokemon with id {id} is {response.json()['name']}"

When a coroutine is called like how a function is called, it returns a Coroutine object. The returned object is still pending scheduling, hence the code isn’t executed at this point. If the coroutine does not get a chance to run, e.g. when the function calling the coroutine exits, dropping the reference to it.

The quickest way to fix it is prefixing an await before it. However, like we discussed earlier, if a function contains an await, then it has to be defined as a coroutine with async. Therefore, going asynchronous usually means an all-in commitment. This also means porting code part-by-part to be asynchronous takes enormous effort.

Using await is a way to schedule a coroutine to the event loop. The behaviour is similar to synchronous programming, it will attempt to start the execution as soon as possible, and will only execute the next line after a result is returned. But there are times we want to have more control, and that’s achieved by turning it to a task.

Tasks: Scheduling and Managing Coroutines

Photo by airfocus on Unsplash

A task can be created, by passing in the coroutine object to asyncio.create_task. It returns a asyncio.Task object referencing the scheduled coroutine. Some control is offered through the returned Task object, for example we can check if it is done, cancel the execution, or set a callback when done.

 task = asyncio.create_task(dex(6))

Photo by Halfcut Pokemon on Unsplash

Unlike the await keyword, you can call asyncio.create_task even in a function, as long as an event loop is available and running. However, being an awaitable object, you can also block execution for subsequent statements with an await too, in a coroutine. However, it is typically used when you only want to revisit the state of execution later (or forget about it).

async def foo() -> str:
    task = asyncio.create_task(dex(104)

    # proceed with other work

    return await task

Photo by Branden Skeli on Unsplash

Futures: Representing Asynchronous Results

Photo by Zulfa Nazer on Unsplash

Next, we have Futures, which is the parent class of the Task we just discussed. I personally find this tricky to explain, but while preparing for the article, I was informed that a future works the way a callback does (still remember?). Let’s say I have a very badly defined function to calculate nth Fibonacci number to simulate a CPU-bound operation

def fibonacci(nth: int) -> int:
    assert nth > 0

    result = ()

    for i in count(1):
        match i:
            case 1:
                result = (0,)

            case 2:
                result += (1,)

            case _:
                result = result[1:] + (sum(result),)

    assert len(result) > 0

    return result[-1]

Asynchronous programming is useful in scenario with a lot of I/O operations. It ensures concurrency by making it possible for CPU-bound operations to work, while waiting for I/O. However, if a CPU-bound operation is taking forever, it would end up stalling everything else. Hence, it is advisible to throw the work to another thread or process. We can use loop.run_in_executor to create an asyncio.Future object to offload the work to another thread.

future = (
    asyncio.get_running_loop()
        .run_in_executor(None, partial(fibonacci, 1500))
)

The first argument can take either a ThreadPoolExecutor or a ProcessPoolExecutor, leave it as None to use the default ThreadPoolExecutor. The second argument takes a callable, we could have rewrite fibonacci with __call__ dunder as discussed in the previous article, but using functools.partial is fine.

Alternatively we can use asyncio.to_thread which is an helpful abstraction to avoid creating a Future ourselves. However, it returns a coroutine, so it has to be awaited, or scheduled as a Task via asyncio.create_task. Also, unlike loop.run_in_executor, we do not get to choose between a ThreadPoolExecutor or ProcessPoolExecutor.

coro = asyncio.to_thread(fibonacci, 1000)

# Schedule it for execution
asyncio.create_task(coro)

# Or await directly
await coro

But we haven’t show how callbacks are possible with Futures. Let’s write a simpler example on this topic. Suppose we have a coroutine that has to be run concurrently in the background, streaming a file content over the internet, and we are interested in finding the line number of a specific line in it.

async def scan_huge_file(future, need_this_line):
    for line_num, line in enumerate(some_huge_file()):
        if line is need_this_line:
            future.set_result(line_num)

Then, when we schedule the tasks (again, we will dive deeper on this in the next article), we would have done something like this.

loop = asyncio.get_running_loop()
future = loop.create_future()

asyncio.create_task(scan_huge_file(future, 'very critical line'))

# do other high-priority tasks
...

# when we are finally done with other tasks
number = await future

The callback is done in scan_huge_file, through future.set_result. The function scheduling the coroutine, only checks if a line number is found, after it is done with other tasks. Of course we could have save a reference to the asyncio.create_task, and then make scan_huge_file return the line number so we can await the task, but this is just to show how a callback can be implemented.

Now we know the three Awaitables, it is time to revisit what it is. As you may have guessed, it is just a term to describe these three types of objects, where we can use await to get the result.

Python vs. JavaScript: Comparison

Photo by Oleksandr Chumak on Unsplash

Asynchronous programming in JavaScript modelled the problem very differently, and hence we have a complete different design. We stopped at Promise just now, and we briefly discussed that because it is the root of the implementation.

AsyncIO is doing a good job abstracting the event loop away from the users. For instance, we only started to call event loop methods, when we were discussing about Futures. The development experience is getting so much better that using await directly to the coroutine, or to the Task (which wraps a coroutine) should be sufficient for most cases.

OTOH, event loop in JavaScript, at least in browser, is almost completely abstracted away from us.

A friend came to me a few days ago, complaining how he is confused about asynchronous programming in the language. Why certain asynchronous functions were called with an await but some without. That prompted me to do a quick check on the topic, and I was quite surprised with the discovery. It turns out if I call an asynchronous function without using await, it will be scheduled in the background.

The discussion session was the main inspiration to this article, sans the Python part. I started with how event handlers led to creation of callbacks, Promise, and ultimately joining the async-await family. Similar to Python, if a function contains any async or await statement in it, then it has to be defined as async.

While on the surface both share similar syntax, but the underlying implementations are completely different. In Python asynchronous was designed around coroutines, while in JavaScript was through Promise. I hope by weaving examples from both language it helps to clarify doubts on the matter.

Looking Ahead: More AsyncIO

Photo by Andrew Ly on Unsplash

Originally I was about to write on schedule and manage tasks, alongside notes on error handling through a fictional project. However, after planning out the content, just Awaitables alone would be long enough for the week. Hence, we would start building a fictional project next week, and discuss the above mentioned topics. Feel free to subscribe to the email newsletter so you don’t miss it, and leave me messages if I have missed anything today.

Thanks for reading, and I shall write again, next week.

For transparency, I’d like to acknowledge that while the code and technical explanations presented in this article are entirely my own, I received editorial assistance from a large language model. This collaboration has helped refine the clarity and structure of my writing. If you’re interested in discussing project collaborations or job opportunities, please feel free to reach out to me here on Medium or via my LinkedIn profile. The article was originally published to Medium.