Coder Legion - Recent posts tagged concurrency

How to Lock a Function Across Processes with a Decorator in Python

Thu, 08 Oct 2026 08:35:18 +0000

Sometimes several processes can call the same function at once, even though its logic is completely unsuited to that. One worker changes the password for an external system, another does exactly the same thing, and a third is trying to log in. Everyone meant well, but now nobody knows the current password XD.

A similar problem comes up when refreshing shared data, generating the same report, or modifying a file. If these operations get in each other's way, we get a classic race condition: a race where the winner sometimes turns out to be an error message.

I wanted to solve this with a decorator that lets you choose between two behaviors:

skip — if someone is already executing the function, skip the next call.
wait — wait until the function is available, then execute it.

I also needed an optional delay after acquiring a lock that had previously been held. Some external systems need a moment to digest a change. Apparently, they enjoy coffee breaks too.

The Decorator

For locking, I used fcntl.flock, which lets you place an operating system lock on an open file. The fcntl module is available on Unix systems, so this example is primarily intended for Linux. The type parameter syntax requires Python 3.12 or later.

Here's the complete implementation:

import fcntl
from collections.abc import Callable
from functools import wraps
from inspect import getfile
from pathlib import Path
from tempfile import gettempdir
from time import sleep
from typing import Literal


def lock_function[**P, R](
    mode: Literal["skip", "wait"] = "skip",
    delay: float = 0,
) -> Callable[[Callable[P, R]], Callable[P, R | None]]:
    def decorator(func: Callable[P, R]) -> Callable[P, R | None]:
        source = str(Path(getfile(func)).resolve()).replace("/", "_").replace(".py", "")
        filename = f"{source}_{func.__name__}.lock"
        lock_path = Path(gettempdir()) / "locks" / filename

        @wraps(func)
        def wrapper(*args: P.args, **kwargs: P.kwargs) -> R | None:
            lock_path.parent.mkdir(parents=True, exist_ok=True)
            with lock_path.open("a") as lock_file:
                try:
                    fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
                except BlockingIOError:
                    if mode == "skip" and delay == 0:
                        return None
                    fcntl.flock(lock_file, fcntl.LOCK_EX)
                    contended = True
                else:
                    contended = False

                try:
                    if contended:
                        sleep(delay)
                        if mode == "skip":
                            return None
                    return func(*args, **kwargs)
                finally:
                    fcntl.flock(lock_file, fcntl.LOCK_UN)

        return wrapper

    return decorator

How to Use It

When another concurrent call is unnecessary, the default skip mode is enough:

@lock_function()
def refresh_shared_cache() -> None:
    ...

The first process acquires the lock and refreshes the data. If the second encounters a lock that's already held, it skips the function body and receives None.

You can also add a delay:

@lock_function(mode="skip", delay=60)
def change_password() -> None:
    ...

Here, a call that encounters a lock that's already held waits to acquire it, waits another 60 seconds, and only then returns without executing the function. This behavior is intentional: the process resumes its other work after the competing operation has finished and the extra pause has elapsed.

If every call should be executed, choose wait:

@lock_function(mode="wait", delay=2)
def update_shared_file(value: str) -> None:
    ...

Suppose three processes try to run the function. The first executes it immediately. The other two wait. Once the lock is released, one of them acquires it, waits two seconds, and executes the function. Then it's the last process's turn.

All three calls will execute, but one at a time. Don't assume they'll run in the order they arrived, though — a lock isn't a queue with numbered tickets.

delay applies only when the first attempt to acquire the lock fails. If the lock was free, the function starts without a delay. This parameter isn't a timeout either: in wait mode, a process can wait for as long as the lock remains held.

How It All Works

A Shared Lock File

When decorating the function, I first determine the file path:

source = str(Path(getfile(func)).resolve()).replace("/", "_").replace(".py", "")
filename = f"{source}_{func.__name__}.lock"
lock_path = Path(gettempdir()) / "locks" / filename

getfile() returns the function's location, and resolve() produces an absolute path. I replace slashes with underscores and append the function name to produce a filename in the shared temporary directory.

This means functions with the same name in different files will usually get separate locks. Call arguments don't affect the name: update_shared_file("a") and update_shared_file("b") compete for the same lock.

Wrapping the Original Function

decorator takes a function, and wrapper replaces it when it's called. Inside wrapper, we acquire the lock and, if appropriate, run the original body:

return func(*args, **kwargs)

@wraps(func) preserves the function's metadata, while the type parameters P and R describe its arguments and return value. The decorated function can also return None, because skip mode allows execution to be skipped.

Attempting to Acquire the Lock

On each call, I create the directory if it's missing and open the file. Then I try to acquire the lock:

fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)

LOCK_EX means an exclusive lock, and LOCK_NB disables waiting. If the lock is already held, I get a BlockingIOError. I then either skip the call immediately or retry without LOCK_NB, this time waiting for access. The flock documentation describes these flags in detail.

The contended variable records whether the first attempt encountered a lock that was already held.

Pausing and Executing

After acquiring a lock that was previously held, I run sleep(delay) and then either skip the function or execute it, depending on the mode.

I hold the lock throughout the pause. This prevents another process from jumping ahead of me while I wait.

Releasing the Lock

Finally, the finally block runs:

finally:
    fcntl.flock(lock_file, fcntl.LOCK_UN)

The lock is released even if the function raises an exception. The exception itself propagates to the caller, and with closes the file.

I don't delete the file. Its existence doesn't mean the lock is held — that's determined by the operating system lock. Deleting and recreating the file could cause processes to lock different files at the same path.

The Limits of This Approach

The decorator handles exceptions, but it isn't prepared for a function getting stuck forever. If a network connection to an external system hangs without a timeout, the code won't reach finally, and the process will keep holding the lock. Subsequent calls in wait mode or skip mode with a delay can then wait forever too. That's why timeouts for these operations need to be configured separately — the decorator guards the door, but it won't rescue anyone from a never-ending phone call.

The processes must see the same lock file. Separate temporary directories in containers or different code locations can mean separate locks. This is a solution for coordinating processes in a shared environment, rather than a ready-made distributed lock.

A similar trap awaits in systemd: a service with PrivateTmp=yes gets its own /tmp and /var/tmp. Two services on the same host, each with a private /tmp, will therefore create different lock files by default, even if the path in the code looks identical. Each politely guards its own door, but those are two different entrances. If they need to block each other, give them a shared lock directory and adjust lock_path accordingly, or deliberately share their private temporary directories through JoinsNamespaceOf=.

All competing calls should also use the decorator. The lock is advisory: code that ignores it can still modify the shared resource. flock won't keep the entire application in check for us.

Summary

A few lines of decorator code are enough to move lock handling out of the function body. skip lets you skip a competing call, wait executes it after acquiring the lock, and delay provides a little extra breathing room after encountering a lock that was already held.

This won't solve every race condition in the project, but at least the processes will stop jostling in the doorway to this one function 😉.

Why Local LLM Inference Needs Admission Control Before It Needs More Hardware

Fri, 11 Sep 2026 02:47:45 +0000

Running an LLM locally is easy to demonstrate.

Start Ollama or vLLM, load a model, send a request, and watch it generate tokens.

The interesting problems start when you send more than one kind of request.

Imagine a local inference server handling two workloads:

Interactive requests where a user is waiting for an answer.
Background jobs generating longer responses that can tolerate delay.

With plenty of spare capacity, both workloads coexist peacefully.

Under contention, they don't.

A few large background requests can occupy enough inference capacity that interactive requests wait behind work that was never latency-sensitive in the first place.

At that point, adding another HTTP worker or increasing a concurrency limit doesn't necessarily solve the problem.

You have a scheduling problem.

Concurrency Is Not Capacity Management

A common first approach is to put a semaphore around inference:

maximum concurrent requests = 4

That's useful. It prevents unlimited work from entering the system.

But it doesn't answer a more important question:

Which four requests should be allowed to run?

Suppose all four slots are occupied by long-running batch generations.

An interactive request arrives.

The concurrency limit is working exactly as designed, yet the user still waits.

The system protected itself from overload, but it didn't protect the workload that mattered most.

This is where admission control becomes different from simple concurrency limiting.

Concurrency asks:

How much work can execute simultaneously?

Admission control asks:

Which work should consume scarce capacity right now?

Tokens Make the Problem More Interesting

LLM workloads also aren't uniform.

Consider two requests:

Request A
input: 500 tokens
maximum output: 200 tokens

Request B
input: 2,000 tokens
maximum output: 4,000 tokens

Counting both as "one request" hides a large difference in potential resource consumption.

A better admission decision can account for both request concurrency and estimated token footprint.

Conceptually:

estimated footprint = input tokens + maximum output tokens

Before admitting a request, the controller asks whether enough capacity remains for that footprint.

The estimate doesn't need to predict the exact number of generated tokens perfectly. It needs to provide a useful bound for deciding whether admitting more work is safe.

When the request finishes, unused capacity can be returned.

This gives the system another dimension of protection beyond request count alone.

Static Limits Still Leave a Policy Problem

You can go further and reserve capacity.

For example:

Interactive: 3 slots
Batch:       1 slot

Now interactive traffic is protected.

But you've created another problem.

What happens when there are no interactive requests?

If three reserved slots sit unused while batch work waits, utilization suffers.

The obvious improvement is borrowing.

Let batch jobs temporarily consume unused interactive capacity:

Interactive demand low
        ↓
Batch borrows spare capacity
        ↓
Interactive demand returns
        ↓
Borrowed capacity must be restored

That last step is where things become interesting.

Borrowing idle capacity is easy.

Safely reclaiming it is the hard part.

Borrowing Without Restoration Isn't Really a Reservation

Suppose an interactive class is guaranteed three units of capacity.

During an idle period, batch workloads borrow two of them.

Then interactive traffic suddenly returns.

If those batch requests can't be interrupted, the interactive reservation doesn't immediately exist anymore. It exists only after the borrowers finish.

That means the actual guarantee depends on restoration behavior.

A useful reservation policy therefore needs to define more than:

interactive floor = 3

It also needs to answer:

How quickly can that floor become usable again?

This distinction matters especially with inference because some resources can be reclaimed immediately while others cannot.

A controller may stop granting new batch work immediately.

But an upstream inference request already running on a GPU may continue until completion.

So "capacity restored" can mean at least two different things:

The scheduler has restored the entitlement.
The physical resource is actually available again.

Those are not necessarily the same moment.

Testing This Against a Real Local Model

I've been experimenting with this problem while building MoFlux, an admission-control system for shared workloads.

One of the useful milestones was running the controller against a real local Ollama instance instead of only synthetic workloads.

The first result wasn't interesting because MoFlux somehow made the model faster.

It didn't.

The interesting result was that the admission policy could operate in front of real local inference and observe the same contention problem that appears in simulations.

That changes what can be tested.

Instead of asking only:

Does the controller make the correct decision?

we can ask:

What happens to real inference when interactive and batch workloads
compete for the same model?

Now measurements such as these become meaningful:

Time to first token
End-to-end latency
Interactive success rate
Batch throughput
Admission rejection rate
Capacity restoration time
Protected-capacity violations

This is where benchmarks start becoming more useful than raw throughput numbers.

Throughput Is Not Always the Goal

Imagine two configurations.

Configuration A completes 110 requests per minute.

Configuration B completes 100.

At first glance, A wins.

But suppose under A:

interactive latency becomes unpredictable,
user-facing requests frequently miss their SLO,
background work dominates capacity during bursts.

Configuration B might complete fewer total requests while completing substantially more useful requests within their objectives.

That's the difference between throughput and goodput.

For shared inference infrastructure, the question isn't simply:

How many generations can this GPU complete?

A more useful question is:

How much useful work can this GPU complete while preserving the
guarantees of the workloads sharing it?

That is a capacity-allocation problem.

Why This Matters for Local Inference

Local inference is becoming increasingly practical.

Developers are running models on workstations, edge machines, private servers, and internal GPU clusters.

The first optimization instinct is often hardware-oriented:

Need more throughput?
Buy a faster GPU.

Sometimes that's the right answer.

But if the actual problem is uncontrolled contention between workloads, more hardware can simply postpone the same failure mode.

Before scaling hardware, it's worth understanding:

Who is consuming capacity?

How much are they allowed to consume?

Which workloads need protection?

Can idle capacity be borrowed?

How quickly can it be reclaimed?

What happens when demand suddenly changes?

Those questions exist whether your inference server is a laptop running Ollama or a production cluster serving thousands of requests.

The scale changes.

The scheduling problem doesn't.

The Bigger Lesson

A single local LLM request is mostly a model-serving problem.

A shared inference server is a distributed-systems problem.

Once multiple workloads compete for a finite model, GPU, memory budget, or concurrency pool, you need policy.

Rate limiting controls how often work arrives.

Concurrency limiting controls how much work runs simultaneously.

Admission control decides whether the system should accept the work at all, given its current obligations.

And workload isolation decides whose work gets protected when capacity becomes scarce.

You may eventually need another GPU.

But before buying one, send your local model two very different workloads at the same time.

You might discover that the first thing you need isn't more capacity.

It's a better way to decide who gets to use the capacity you already have.

Profiling a Go Service in Production: pprof in 10 Minutes

Mon, 20 Jul 2026 19:45:14 +0000

Go ships with a profiler built in. No third-party tools, no agents to install, no instrumentation to add upfront. If your service imports net/http/pprof, you can profile it live in production right now — CPU usage, memory allocations, goroutine counts, blocking operations. The data is available over HTTP, readable with standard Go tooling.

Most engineers know pprof exists. Fewer have actually used it under pressure, on a real service, to find a real problem. This post walks through the mechanics — how to enable it, how to collect a profile, how to read a flame graph — and then shows the three classes of problems it catches most often.

Enabling pprof

For an HTTP service, one import is all it takes:

import (
    "log"
    "net/http"

    _ "net/http/pprof" // registers handlers on DefaultServeMux
)

func main() {
    // Your service setup...

    // pprof endpoints are now available on DefaultServeMux
    go func() {
        log.Println(http.ListenAndServe("localhost:6060", nil))
    }()

    // ... rest of main
}

The blank import registers the pprof HTTP handlers automatically. They’re available at:

GET /debug/pprof/ — index of available profiles
GET /debug/pprof/goroutine — all current goroutines
GET /debug/pprof/heap — memory allocations
GET /debug/pprof/profile?seconds=30 — CPU profile (30-second sample)
GET /debug/pprof/block — goroutine blocking events
GET /debug/pprof/mutex — mutex contention

Security note: bind pprof to localhost only, or a private interface. Never expose it on your public port. If your service runs in Kubernetes, use kubectl port-forward to reach it.

For services not using DefaultServeMux, register the handlers explicitly:

mux := http.NewServeMux()
mux.HandleFunc("/debug/pprof/", pprof.Index)
mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
mux.HandleFunc("/debug/pprof/trace", pprof.Trace)

Collecting a CPU profile

With the service running under load (profiling a quiet service tells you nothing useful), collect a 30-second CPU sample:

go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

This downloads the profile and drops you into an interactive shell. Or skip the shell and go straight to a flame graph:

# Collect the profile to a file
curl -o cpu.prof "http://localhost:6060/debug/pprof/profile?seconds=30"

# Open the flame graph in a browser
go tool pprof -http=:8080 cpu.prof

The -http flag serves a web UI with multiple views. The flame graph is the most useful starting point.

Reading a flame graph

A flame graph shows where your program spends time. Each horizontal bar is a function. Width represents time — wider means more CPU. The vertical axis is the call stack — functions lower down called the ones above them.

|          processJob          |     |  encode  |
|       fetchFromDB      |other|
|         sql.Query            |
|          net.Read            |

You’re looking for wide bars near the top — those are the functions consuming the most CPU. If json.Marshal is unexpectedly wide, you’re spending a lot of time serializing. If runtime.mallocgc is wide, you’re allocating heavily. If sync.Mutex.Lock is wide, you have lock contention.

The three most common findings in worker pool services:

Problem 1: Excessive allocations

Symptom: runtime.mallocgc is prominent in the CPU profile. Heap profile shows high allocation rate with short-lived objects.

# Collect a heap profile
curl -o heap.prof "http://localhost:6060/debug/pprof/heap"
go tool pprof -http=:8080 heap.prof

In the heap profile, switch to “alloc_objects” or “alloc_space” view (not “inuse” — that shows what’s live, not what’s being allocated). Look for hot allocation sites.

Common cause in worker pools: allocating a new buffer or slice on every job iteration.

// ❌ Allocates a new byte slice for every job
func processJob(ctx context.Context, job Job) error {
    buf := make([]byte, 4096)
    // use buf...
    return nil
}

// ✅ Reuse buffers with sync.Pool
var bufPool = sync.Pool{
    New: func() interface{} {
        b := make([]byte, 4096)
        return &b
    },
}

func processJob(ctx context.Context, job Job) error {
    bufPtr := bufPool.Get().(*[]byte)
    defer bufPool.Put(bufPtr)
    buf := *bufPtr
    // use buf...
    return nil
}

sync.Pool maintains a pool of reusable objects. Objects not retrieved before the next GC cycle are collected, so it doesn’t prevent garbage collection — it reduces the rate of allocation, which reduces GC pressure.

Problem 2: Goroutine leak

Symptom: goroutine count grows over time and never decreases.

# Check current goroutine count
curl "http://localhost:6060/debug/pprof/goroutine?debug=1"

The debug=1 parameter returns a text listing of all goroutines with stack traces. Look for hundreds or thousands of goroutines all blocked at the same location — that’s your leak.

goroutine 1042 [chan receive, 3 minutes]:
main.worker()
    /app/worker.go:45 +0x68
created by main.startWorkers
    /app/pool.go:23 +0x4c

goroutine 1043 [chan receive, 3 minutes]:
...
(997 more like this)

A thousand goroutines blocked on chan receive for 3 minutes means the jobs channel was never closed. The fix is covered in the concurrency mistakes post in this series — the producer must own close(jobs) via defer.

You can also check the goroutine count programmatically:

// In your health check endpoint
func healthHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "goroutines: %d\n", runtime.NumGoroutine())
}

If that number grows continuously over hours of runtime and never stabilizes, you have a leak.

Problem 3: Lock contention

Symptom: throughput is lower than CPU utilization suggests it should be. Workers are running but not making progress.

Mutex profiling is off by default because it has overhead. Enable it from inside the process only while investigating contention:

import "runtime"

runtime.SetMutexProfileFraction(1)
defer runtime.SetMutexProfileFraction(0)

Then collect the mutex profile:

curl -o mutex.prof "http://localhost:6060/debug/pprof/mutex"
go tool pprof -http=:8080 mutex.prof

Common cause in worker pools: a shared results map protected by a single mutex, written to by all workers simultaneously.

// ❌ All workers contend on one mutex
type ResultCollector struct {
    mu      sync.Mutex
    results map[string]Result
}

func (c *ResultCollector) Add(id string, result Result) {
    c.mu.Lock()
    c.results[id] = result
    c.mu.Unlock()
}

Solutions, in order of preference:

// ✅ Option 1: Use a results channel instead of shared state
results := make(chan Result, 1000)

// Workers send to channel (no lock)
go func() {
    for job := range jobs {
        result := processJob(ctx, job)
        results <- result
    }
}()

// Single collector goroutine reads from channel (no contention)
go func() {
    for result := range results {
        collected[result.ID] = result
    }
}()

// ✅ Option 2: sync.Map for concurrent read-heavy workloads
var results sync.Map
results.Store(id, result)
val, ok := results.Load(id)

The channel approach is idiomatic Go — share memory by communicating rather than communicating by sharing memory. The sync.Map approach is better when you need random access to results across goroutines.

A quick reference for which profile to use

High CPU, unclear where — CPU profile: pprof .../profile?seconds=30.

Memory growing over time — heap profile (inuse_space): pprof .../heap.

High GC pressure — heap profile (alloc_objects): pprof .../heap.

Goroutine count growing — goroutine profile: pprof .../goroutine?debug=1.

Low throughput despite low CPU — mutex or block profile: pprof .../mutex or .../block. Enable mutex profiling with runtime.SetMutexProfileFraction; enable block profiling with runtime.SetBlockProfileRate.

One workflow for production incidents

When something is wrong and you need to understand it quickly:

# 1. Check goroutine count (cheap, immediate)
curl localhost:6060/debug/pprof/goroutine?debug=2 | head -50

# 2. If CPU is high, get a 30s CPU profile
curl -o cpu.prof localhost:6060/debug/pprof/profile?seconds=30

# 3. If memory is growing, get a heap snapshot
curl -o heap.prof localhost:6060/debug/pprof/heap

# 4. Open whichever profile is relevant
go tool pprof -http=:8080 cpu.prof

The whole process takes under 5 minutes. You don’t need to reproduce the problem locally, you don’t need to redeploy with extra instrumentation, and you don’t need to restart the service. The profiler runs against the live process.

Summary

pprof is built into Go’s standard library and available with a single import. The three problems it catches most reliably in worker pool services are excessive allocations (fix: sync.Pool), goroutine leaks (fix: close channels, use context), and mutex contention (fix: channel-based result collection or sync.Map). The flame graph view in go tool pprof -http is the fastest path from profile to finding.

The only prerequisite is running the service under realistic load when you collect the profile. A profile from a quiet service tells you where your code could spend time, not where it does.

Previous in this series: context.Context Is Not Optional
Next: Container-aware resource management in Go.

context.Context Is Not Optional: A Practical Guide to Cancellation in Go Services

Mon, 20 Jul 2026 16:58:42 +0000

Every Go service that does I/O — database calls, HTTP requests, queue polling, file reads — should be passing a context.Context through every layer. In practice, a large number of codebases treat context.Context as optional plumbing that gets added later, or as something only the HTTP handler layer needs to worry about.

That’s the mistake. By the time you realize you need cancellation deep in a worker pool, retrofitting it is painful. This post shows what context propagation looks like in a real worker pool, what the failure modes are when you skip it, and how to do it correctly from the start.

What context.Context actually does

A context.Context carries three things:

A cancellation signal — a channel that’s closed when the work should stop
A deadline or timeout — an absolute time after which the context is automatically cancelled
Key-value pairs — request-scoped values like trace IDs (use sparingly)

When a context is cancelled, any blocking operation that’s watching that context should stop and return an error. The context propagates down the call stack — a parent context cancelled cancels all children. It never propagates up.

The critical point: the context has to be passed all the way down to the code that actually blocks. Holding it at the HTTP handler layer and never passing it to your database call means cancellation never reaches the work.

What happens without context

Here is a worker pool that ignores context:

func runWorkers(jobs <-chan Job) {
    var wg sync.WaitGroup
    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for job := range jobs {
                processJob(job) // no context
            }
        }()
    }
    wg.Wait()
}

func processJob(job Job) error {
    result, err := fetchFromDatabase(job.ID) // no context
    if err != nil {
        return err
    }
    return sendToDownstream(result) // no context
}

Now consider what happens when:

The calling service shuts down
A client disconnects
A deadline passes
An operator sends SIGTERM

The workers keep running. fetchFromDatabase keeps waiting. sendToDownstream keeps trying. The process can’t exit cleanly because goroutines are blocked on operations that have no way of knowing they should stop.

In the best case, the process takes 30 seconds to shut down waiting for goroutines to unblock naturally. In the worst case, it never terminates and has to be killed, potentially leaving in-flight work in an inconsistent state.

The correct structure: context flows in, errors flow out

The pattern is simple. Every function that does I/O or blocks takes a context as its first parameter:

func runWorkers(ctx context.Context, jobs <-chan Job) error {
    var wg sync.WaitGroup
    errs := make(chan error, 10)

    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for {
                select {
                case <-ctx.Done():
                    return // context cancelled, stop cleanly
                case job, ok := <-jobs:
                    if !ok {
                        return // channel closed, no more jobs
                    }
                    if err := processJob(ctx, job); err != nil {
                        select {
                        case errs <- err:
                        default:
                        }
                    }
                }
            }
        }()
    }

    wg.Wait()
    close(errs)

    // Return first error if any
    return <-errs
}

func processJob(ctx context.Context, job Job) error {
    result, err := fetchFromDatabase(ctx, job.ID)
    if err != nil {
        return err
    }
    return sendToDownstream(ctx, result)
}

Now when the context is cancelled — whether by timeout, operator signal, or parent cancellation — each worker’s select unblocks on ctx.Done() and returns cleanly. fetchFromDatabase and sendToDownstream propagate the context to their underlying I/O calls (database drivers, HTTP clients), which respect it.

Wiring context to OS signals for clean shutdown

The context that flows into your worker pool should ultimately be rooted at main, wired to OS signals:

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    jobs := make(chan Job, 100)

    // Producer
    go func() {
        defer close(jobs)
        if err := loadJobs(ctx, jobs); err != nil {
            log.Printf("producer stopped: %v", err)
        }
    }()

    // Workers
    if err := runWorkers(ctx, jobs); err != nil {
        log.Printf("workers stopped with error: %v", err)
        os.Exit(1)
    }

    log.Println("shutdown complete")
}

signal.NotifyContext (added in Go 1.16) returns a context that’s cancelled when SIGINT or SIGTERM arrives. That cancellation propagates through loadJobs and runWorkers automatically. When the signal arrives, the producer stops generating new jobs, in-flight I/O observes cancellation through the propagated context, workers exit, and the process shuts down cleanly.

Timeouts: per-job vs. pool-wide

There are two distinct timeout requirements in a worker pool, and they need separate contexts:

Pool-wide timeout: how long the entire batch is allowed to run. Use context.WithTimeout or context.WithDeadline on the root context.

Per-job timeout: how long a single job is allowed to take. Use a derived context inside the worker loop.

func runWorkers(ctx context.Context, jobs <-chan Job) error {
    // ctx already carries the pool-wide deadline from the caller

    var wg sync.WaitGroup
    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for {
                select {
                case <-ctx.Done():
                    return
                case job, ok := <-jobs:
                    if !ok {
                        return
                    }

                    // Per-job timeout: 30 seconds max per job
                    jobCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
                    err := processJob(jobCtx, job)
                    cancel() // always call cancel to release resources

                    if err != nil {
                        log.Printf("job %s failed: %v", job.ID, err)
                    }
                }
            }
        }()
    }

    wg.Wait()
    return ctx.Err()
}

Two things to note. First, jobCtx is derived from ctx — if the pool-wide context is cancelled, the per-job context is also cancelled automatically. The hierarchy works in your favor. Second, cancel() is called immediately after processJob returns, not deferred. Deferring a context cancel inside a loop means you accumulate uncancelled contexts until the goroutine exits. Call cancel immediately after the work is done.

Checking context in CPU-bound work

The ctx.Done() pattern works for I/O that blocks, because the underlying library propagates the context. But what about CPU-bound work — a job that does heavy computation without any I/O?

func processJob(ctx context.Context, job Job) error {
    chunks := split(job.Data)

    for i, chunk := range chunks {
        // Check context periodically in long-running CPU work
        if i%100 == 0 {
            select {
            case <-ctx.Done():
                return ctx.Err()
            default:
            }
        }

        if err := processChunk(chunk); err != nil {
            return err
        }
    }

    return nil
}

The select with a default branch is non-blocking — it checks whether the context is done and moves on immediately if not. Checking every 100 iterations (or every N milliseconds using a ticker) is a reasonable balance between responsiveness and overhead.

The three rules

1. Context is always the first parameter.

func doWork(ctx context.Context, ...) error

Not a struct field, not a global, not injected via closure. First parameter, every time. This is a Go convention strong enough to be in the standard library style guide.

2. Never store context in a struct.

// ❌
type Worker struct {
    ctx context.Context
}

// ✅
func (w *Worker) Process(ctx context.Context, job Job) error

Contexts are request-scoped. A struct outlives any single request. Storing context in a struct means you’re using the wrong context for future requests, or holding a cancelled context longer than you should.

3. Always call cancel.

ctx, cancel := context.WithTimeout(parent, 5*time.Second)
defer cancel() // even if the timeout fires first

If the timeout fires before you call cancel, calling cancel is a no-op. But if the work finishes before the timeout, calling cancel releases the timer resources immediately rather than waiting for the timeout to expire. It costs nothing and prevents a resource leak.

Summary

Clean shutdown on SIGTERM — signal.NotifyContext at main, propagate down.

Stop workers on cancellation — select { case <-ctx.Done(): return } in the worker loop.

Per-job timeout — context.WithTimeout(ctx, duration) inside the loop, cancel immediately after.

CPU-bound work — periodic select { case <-ctx.Done(): return ctx.Err(); default: }.

Function signatures — ctx context.Context is always the first parameter.

Context propagation is not boilerplate. It is the mechanism that lets your service behave correctly under pressure — when deadlines pass, when clients disconnect, when operators need to restart the service without waiting for goroutines to unblock naturally. The cost of adding it from the start is a few extra parameters. The cost of retrofitting it later is much higher.

Previous in this series: Three Go Concurrency Mistakes I See in Almost Every Worker Pool
Next: Profiling a Go service in production with pprof.

Why Modern Engineering Teams Must Prioritize System Observability Over Raw Performance

Wed, 15 Jul 2026 14:45:00 +0000

Engineering teams often chase performance: faster APIs, lower latency, optimized queries, reduced CPU usage. But in 2026, performance alone is no longer the defining factor of a reliable system. Modern architectures — distributed, event‑driven, containerized, multi‑service — demand something far more fundamental: [system observability][1].

Performance tells you how fast your system is.
Observability tells you why it behaves the way it does.

And for engineering teams, especially small ones, this difference determines whether a system scales or collapses under pressure.

Performance Without Observability Is a Blindfold
A fast system can still be fragile.
A highly optimized service can still fail silently.
A low‑latency API can still degrade unpredictably under load.

When teams focus exclusively on performance metrics, they optimize symptoms instead of causes. They see spikes, drops, delays — but they don’t see the internal relationships that create them.

Observability exposes:

hidden dependencies

cascading failures

slow internal paths

unexpected retry storms

misconfigured backoff logic

silent timeouts

resource starvation

cross‑service bottlenecks

Without this visibility, engineers operate reactively. They fix what they see, not what actually breaks the system.

The Three Pillars of Observability
Modern observability is not just logs. It is a structured, multi‑layered system that allows engineers to understand why something happened.

Metrics — the heartbeat of the system
Metrics show trends, anomalies, and thresholds.
They reveal when something is off — but not why.

Key engineering metrics include:

request latency distribution

error rate spikes

queue depth growth

retry counts

CPU throttling events

memory pressure

cache hit ratios

Metrics are the first signal, not the full story.

Logs — the narrative
Logs provide context.
They explain the events behind the metrics.

But logs alone are insufficient.
Without structure, correlation, and context, logs become noise.

Modern teams rely on:

structured JSON logs

correlation IDs

distributed trace IDs

log sampling

log enrichment with metadata

Logs transform raw events into actionable insights.

Traces — the map of the system
Distributed tracing is the most powerful observability tool of 2026.

Traces show:

how a request travels through the system

where it slows down

where it fails

where retries accumulate

where bottlenecks form

where unexpected dependencies appear

Traces reveal the architecture as it actually behaves — not as it was designed.

Why Observability Matters More for Small Teams
Large companies have SRE departments, dedicated monitoring teams, and 24/7 on‑call rotations.
Small engineering teams do not.

For small teams, observability is not a luxury — it is a multiplier.

It allows them to:

detect issues before users notice

reduce on‑call stress

shorten incident resolution time

prevent cascading failures

scale without rewriting the system

understand real‑world behavior instead of assumptions

Observability replaces guesswork with clarity.

Observability Enables Predictable Engineering
Predictability is the most valuable trait of a modern system.

A predictable system:

fails gracefully

recovers automatically

scales linearly

behaves consistently under load

provides clear signals when something is wrong

Observability is the foundation of this predictability.
It transforms engineering from reactive firefighting into proactive system design.

The Future: Observability as a Core Engineering Skill
In 2026, observability is no longer a DevOps specialty.
It is a core engineering skill.

Teams that master observability:

ship faster

debug faster

scale safer

maintain stability

reduce operational cost

build systems that survive real‑world conditions

Performance matters.
But system observability determines whether performance is sustainable.

Building a Concurrency-Safe Movie Reservation System

Fri, 10 Jul 2026 07:50:58 +0000

Building a Concurrency-Safe Movie Reservation System with FastAPI, Redis, and PostgreSQL

Introduction

One of the most interesting challenges in backend engineering isn't building CRUD APIs—it's designing systems that continue to behave correctly when hundreds of users interact with them simultaneously.

Consider this scenario:

A blockbuster movie opens for booking.

100 users click Reserve for the exact same seat at the exact same millisecond.

Without proper concurrency control, multiple users could end up paying for the same seat. Preventing this is one of the core responsibilities of a reservation system.

To better understand how production booking systems solve this problem, I built a Concurrency-Safe Movie Reservation Backend using FastAPI, PostgreSQL, Redis, and SQLAlchemy.

The goal wasn't to recreate BookMyShow or Ticketmaster. Instead, it was to explore the backend engineering concepts that make these systems reliable under concurrent traffic.

The Problem

At first glance, reserving a seat seems straightforward:

Check if the seat is available.
Create a reservation.
Mark the seat as booked.

The problem appears when multiple requests arrive simultaneously.

Imagine two requests reaching the server at nearly the same time.

Request A
Check seat availability
Seat is free

Request B
Check seat availability
Seat is free

Request A
Reserve seat

Request B
Reserve seat

Both requests observed the seat as available before either completed the reservation.

This classic race condition can result in duplicate bookings if the system isn't designed carefully.

Designing the Reservation Workflow

To prevent this, I designed the reservation flow with multiple layers of protection rather than relying on a single mechanism.

The workflow looks like this:

User requests available seats
        ↓
Requested seats are temporarily locked in Redis
        ↓
Lock ownership is assigned
        ↓
Reservation request is submitted
        ↓
Lock ownership is verified
        ↓
Reservation is created inside a PostgreSQL transaction
        ↓
Reserved seats are persisted
        ↓
Redis locks are released

Each layer addresses a different failure scenario.

Layer 1: Distributed Seat Locking with Redis

The first challenge is preventing multiple users from selecting the same seat simultaneously.

Redis is ideal for this because it provides extremely fast in-memory operations.

Each seat lock is stored using a key similar to:

lock:showtime:{showtime_id}:{seat_label}

Every lock contains:

User ID
Expiration timestamp

These locks are temporary.

If a user abandons the booking process, the lock eventually expires, allowing someone else to reserve the seat.

Preventing Partial Seat Locking

Suppose a user wants three seats.

If the application locks seats individually, this situation could occur:

Seat A ✓
Seat B ✓
Seat C ✗

Now the user owns only part of the requested seats.

This creates unnecessary complexity.

Instead, I used Redis Lua scripts so the locking operation becomes atomic.

Either:

every requested seat is locked

none of them are.

This eliminates inconsistent intermediate states.

Lock Ownership Verification

A Redis lock alone isn't enough.

Imagine this sequence:

User A acquires a seat lock.
The lock expires.
User B acquires the same seat.
User A submits an old reservation request.

Without ownership validation, the outdated request could incorrectly succeed.

Before creating a reservation, the application verifies that every requested seat is still locked by the same user.

If ownership has changed, the reservation is rejected.

Database Constraints as the Final Safety Net

Redis helps coordinate requests, but PostgreSQL remains the source of truth.

Even if every application-level protection somehow failed, the database still guarantees consistency.

A unique constraint ensures that a seat cannot be reserved twice for the same showtime.

This means duplicate reservations are impossible at the persistence layer.

Transactional Reservation Processing

Creating a reservation isn't a single database operation.

The application needs to:

Create the reservation
Insert reserved seats
Update related records
Persist everything consistently

If one operation fails midway, the database should not be left in a partially updated state.

For that reason, reservation creation happens inside a single PostgreSQL transaction.

Either:

every operation succeeds

everything rolls back.

This guarantees atomicity and consistency.

Idempotent Reservation Requests

Real users retry requests.

Browsers retry requests.

Mobile networks retry requests.

Without idempotency, duplicate retries may accidentally create multiple reservations.

To solve this, reservation requests include an Idempotency Key.

If the same request is received multiple times, the server returns the original result instead of creating another reservation.

This makes retries safe.

Protecting the API with Rate Limiting

Reservation endpoints are attractive targets for abuse.

To prevent excessive requests, I implemented Redis-based rate limiting using simple atomic operations:

INCR
EXPIRE

This throttles repeated requests while keeping the implementation lightweight.

Background Workers

Some operations shouldn't happen during the request-response cycle.

Background workers periodically:

Release expired Redis locks
Expire abandoned reservations
Restore seat availability

This keeps reservation state consistent without blocking user requests.

Stress Testing the System

After implementing the concurrency protections, I wanted to verify that they actually worked.

Using Locust, I simulated:

100 concurrent users
All attempting to reserve the exact same seat
At nearly the same instant

The results were exactly what I hoped for.

Reservation Attempts : 100

Successful Reservations : 1

Failed Reservations : 99

Duplicate Bookings : 0

Exactly one reservation succeeded.

Every competing request failed safely.

I also tested:

Multiple users reserving different seats simultaneously
Redis lock expiration
Database transaction failures
Redis failure scenarios

These tests increased my confidence that the reservation workflow behaves correctly under concurrent load.

What I Learned

Building this project taught me much more than another CRUD application ever could.

I gained practical experience with:

Race conditions
Distributed locking
Database transactions
Concurrency control
Idempotent APIs
Background processing
Redis coordination
Transactional consistency
Backend architecture

More importantly, it changed the way I think about backend development.

Correctness under concurrency is often more important than adding new features.

A system that behaves predictably under load is significantly harder—and more rewarding—to build.

Final Thoughts

This project was an opportunity to move beyond simple REST APIs and explore problems that real booking platforms face every day.

There are still many areas I'd like to improve, including:

Optimistic locking strategies
Distributed tracing
Event-driven reservation workflows
Horizontal scaling
Kubernetes deployment
Observability with Prometheus and Grafana

If you're interested in backend engineering or distributed systems, I'd love to hear how you would approach these challenges differently.

Constructive feedback and architectural discussions are always welcome.

Enjoyed this article? I also shared a shorter version on LinkedIn where I'm collecting architectural feedback from backend engineers. I'd love to hear your thoughts there as well.

LinkedIn Discussion: https://www.linkedin.com/posts/rahul-ch-434b1a250_backend-python-fastapi-ugcPost-7480908963971166208-KY2M/

Happy building! 🚀

Rust Concurrency: Threads, Channels, Mutex & Sync (Part 4)

Wed, 27 May 2026 07:36:28 +0000

title: Rust Concurrency: Threads, Channels, Mutex & Sync (Part 4)
published: true
tags: rust, concurrency, intermediate, systems
cover_image: https://images.unsplash.com/photo-1558494949-ef010cbdcc31?w=1000

series: Core Rust Concepts

This is Part 4 of the Core Rust Concepts series.

Part 1 — Ownership, Borrowing, Lifetimes, Traits, Result/Option, Pattern Matching

Part 2 — Closures, Iterators, Generics, Enums, Smart Pointers, Async/Await

Part 3 — Macros, Modules, Testing, Unsafe Rust, FFI

Threads
Message Passing with Channels
Send and Sync Traits
Mutex and RwLock
Atomics
Building a CLI Tool with clap

19. Threads

Rust's standard library provides OS threads via std::thread. The ownership and type system prevents data races at compile time — if your code compiles, it's free of data races. That's a guarantee no other systems language makes.

Spawning a thread

use std::thread;
use std::time::Duration;

fn main() {
    let handle = thread::spawn(|| {
        for i in 1..=5 {
            println!("thread: {i}");
            thread::sleep(Duration::from_millis(50));
        }
    });

    for i in 1..=3 {
        println!("main:   {i}");
        thread::sleep(Duration::from_millis(80));
    }

    handle.join().unwrap(); // wait for the thread to finish
}

Moving data into a thread

Use move to transfer ownership of data into the thread closure:

use std::thread;

fn main() {
    let data = vec![1, 2, 3, 4, 5];

    let handle = thread::spawn(move || {
        // data is moved into this thread
        let sum: i32 = data.iter().sum();
        println!("sum: {sum}");
    });

    // println!("{:?}", data); ← compile error: data was moved

    handle.join().unwrap();
}

Spawning many threads

use std::thread;

fn main() {
    let handles: Vec<_> = (0..8)
        .map(|i| {
            thread::spawn(move || {
                println!("worker {i} done");
                i * i
            })
        })
        .collect();

    let results: Vec<_> = handles
        .into_iter()
        .map(|h| h.join().unwrap())
        .collect();

    println!("squares: {:?}", results);
}

thread::spawn returns a JoinHandle<T> where T is the return type of the closure. Always .join() your handles — a panicking thread silently fails otherwise.

20. Message Passing with Channels

Rust's standard library includes multiple-producer, single-consumer (mpsc) channels. The philosophy: "do not communicate by sharing memory; share memory by communicating."

Basic channel

use std::sync::mpsc;
use std::thread;

fn main() {
    let (tx, rx) = mpsc::channel();

    thread::spawn(move || {
        let msgs = vec!["hello", "from", "the", "thread"];
        for msg in msgs {
            tx.send(msg).unwrap();
        }
    });

    for received in rx {
        println!("got: {received}");
    }
}

Multiple producers

use std::sync::mpsc;
use std::thread;

fn main() {
    let (tx, rx) = mpsc::channel::<String>();

    for id in 0..4 {
        let tx_clone = tx.clone(); // each thread gets its own sender
        thread::spawn(move || {
            tx_clone.send(format!("msg from worker {id}")).unwrap();
        });
    }

    drop(tx); // drop the original so rx knows when all senders are gone

    for msg in rx {
        println!("{msg}");
    }
}

Synchronous channel (bounded)

use std::sync::mpsc;
use std::thread;

fn main() {
    // Buffer of 2 — sender blocks when full
    let (tx, rx) = mpsc::sync_channel::<i32>(2);

    thread::spawn(move || {
        for i in 0..5 {
            println!("sending {i}");
            tx.send(i).unwrap(); // blocks at i=2 until receiver reads
        }
    });

    thread::sleep(std::time::Duration::from_millis(200));

    for val in rx {
        println!("received {val}");
    }
}

For more advanced channel patterns (broadcast, watch, oneshot), use the tokio::sync or crossbeam-channel crates.

21. Send and Sync Traits

These two marker traits are the foundation of Rust's fearless concurrency. You rarely implement them manually — the compiler derives them automatically.

Trait	Meaning
`Send`	Safe to transfer ownership to another thread
`Sync`	Safe to share a reference across threads (`&T` is `Send`)

use std::thread;
use std::rc::Rc;
use std::sync::Arc;

fn main() {
    // Rc<T> is NOT Send — can't move across threads
    // let rc = Rc::new(42);
    // thread::spawn(move || println!("{}", rc)); ← compile error!

    // Arc<T> IS Send — atomic reference counting
    let arc = Arc::new(42);
    let arc_clone = Arc::clone(&arc);

    thread::spawn(move || {
        println!("thread sees: {}", arc_clone);
    }).join().unwrap();

    println!("main sees: {}", arc);
}

Types and their Send/Sync status:

Type	Send	Sync
`i32`, `f64`, `bool`	✅	✅
`String`, `Vec<T>`	✅	✅
`Rc<T>`	❌	❌
`Arc<T>`	✅	✅
`RefCell<T>`	✅	❌
`Mutex<T>`	✅	✅
`*mut T` (raw pointer)	❌	❌

If you try to share a non-Sync type across threads, the compiler refuses to compile. No runtime surprises.

22. Mutex and RwLock

When multiple threads need to mutate shared data, use a Mutex (mutual exclusion lock). Wrap it in Arc to share ownership across threads.

Mutex

use std::sync::{Arc, Mutex};
use std::thread;

fn main() {
    let counter = Arc::new(Mutex::new(0));
    let mut handles = vec![];

    for _ in 0..10 {
        let counter = Arc::clone(&counter);
        let handle = thread::spawn(move || {
            let mut num = counter.lock().unwrap(); // blocks until lock acquired
            *num += 1;
        }); // lock is released here when `num` drops
        handles.push(handle);
    }

    for h in handles { h.join().unwrap(); }

    println!("result: {}", *counter.lock().unwrap()); // 10
}

RwLock — many readers, one writer

use std::sync::{Arc, RwLock};
use std::thread;

fn main() {
    let data = Arc::new(RwLock::new(vec![1, 2, 3]));

    // Spawn 4 reader threads
    let readers: Vec<_> = (0..4).map(|i| {
        let data = Arc::clone(&data);
        thread::spawn(move || {
            let r = data.read().unwrap(); // multiple readers allowed
            println!("reader {i}: {:?}", *r);
        })
    }).collect();

    // One writer thread
    let data_w = Arc::clone(&data);
    let writer = thread::spawn(move || {
        let mut w = data_w.write().unwrap(); // exclusive access
        w.push(4);
        println!("writer pushed 4");
    });

    for r in readers { r.join().unwrap(); }
    writer.join().unwrap();
}

Mutex vs RwLock:

	`Mutex<T>`	`RwLock<T>`
Readers	One at a time	Many simultaneously
Writers	One at a time	One at a time (exclusive)
Best for	Write-heavy workloads	Read-heavy workloads
Overhead	Lower	Higher

Avoiding deadlocks

use std::sync::{Arc, Mutex};

fn main() {
    let a = Arc::new(Mutex::new(1));
    let b = Arc::new(Mutex::new(2));

    // ✅ Always acquire locks in the same order
    {
        let _lock_a = a.lock().unwrap();
        let _lock_b = b.lock().unwrap();
        println!("safe");
    }

    // ❌ Acquiring in different orders across threads = deadlock risk
    // thread 1: lock a → lock b
    // thread 2: lock b → lock a  ← deadlock!
}

23. Atomics

For simple counters and flags shared between threads, std::sync::atomic types are faster than a Mutex — they use hardware atomic instructions with no locking.

use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::thread;

fn main() {
    let counter = Arc::new(AtomicUsize::new(0));
    let mut handles = vec![];

    for _ in 0..8 {
        let counter = Arc::clone(&counter);
        handles.push(thread::spawn(move || {
            for _ in 0..1000 {
                counter.fetch_add(1, Ordering::Relaxed);
            }
        }));
    }

    for h in handles { h.join().unwrap(); }
    println!("total: {}", counter.load(Ordering::SeqCst)); // 8000
}

Memory ordering cheat sheet

Ordering	Use case
`Relaxed`	Counters — no ordering guarantees needed
`Acquire`	Load that synchronizes with a `Release` store
`Release`	Store that publishes data to other threads
`AcqRel`	Read-modify-write (e.g. `fetch_add` with sync)
`SeqCst`	Strongest — total order across all threads

use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::thread;

fn main() {
    let ready = Arc::new(AtomicBool::new(false));
    let ready_clone = Arc::clone(&ready);

    let worker = thread::spawn(move || {
        thread::sleep(std::time::Duration::from_millis(100));
        ready_clone.store(true, Ordering::Release);
        println!("worker: done");
    });

    // Spin-wait (not ideal in prod — use a Condvar or channel instead)
    while !ready.load(Ordering::Acquire) {
        thread::yield_now();
    }
    println!("main: worker finished");
    worker.join().unwrap();
}

Available atomic types: AtomicBool, AtomicI8/16/32/64, AtomicU8/16/32/64, AtomicUsize, AtomicIsize, AtomicPtr<T>.

24. Building a CLI Tool with clap

clap is the standard Rust library for building command-line interfaces. It gives you argument parsing, help text, and subcommands with minimal boilerplate.

# Cargo.toml
[dependencies]
clap = { version = "4", features = ["derive"] }

Basic CLI

use clap::Parser;

/// A simple file word counter
#[derive(Parser, Debug)]
#[command(name = "wordcount")]
#[command(about = "Count words in a string or file", long_about = None)]
struct Cli {
    /// The text to count words in
    #[arg(short, long)]
    text: Option<String>,

    /// Show character count too
    #[arg(short, long, default_value_t = false)]
    chars: bool,

    /// Verbosity level
    #[arg(short, long, action = clap::ArgAction::Count)]
    verbose: u8,
}

fn main() {
    let cli = Cli::parse();

    let input = cli.text.unwrap_or_else(|| String::from("hello world"));
    let words = input.split_whitespace().count();

    if cli.verbose > 0 {
        println!("Input: {:?}", input);
    }

    println!("Words: {words}");

    if cli.chars {
        println!("Chars: {}", input.chars().count());
    }
}

Running it:

$ cargo run -- --text "the quick brown fox" --chars -v
Input: "the quick brown fox"
Words: 4
Chars: 19

$ cargo run -- --help
Count words in a string or file

Usage: wordcount [OPTIONS]

Options:
  -t, --text <TEXT>   The text to count words in
  -c, --chars         Show character count too
  -v, --verbose...    Verbosity level
  -h, --help          Print help

Subcommands

use clap::{Parser, Subcommand};

#[derive(Parser)]
#[command(name = "mytool", about = "A multi-command CLI")]
struct Cli {
    #[command(subcommand)]
    command: Commands,
}

#[derive(Subcommand)]
enum Commands {
    /// Add two numbers
    Add { a: f64, b: f64 },
    /// Multiply two numbers
    Mul { a: f64, b: f64 },
    /// Greet someone
    Greet {
        name: String,
        #[arg(short, long, default_value = "Hello")]
        greeting: String,
    },
}

fn main() {
    let cli = Cli::parse();

    match cli.command {
        Commands::Add { a, b } => println!("{} + {} = {}", a, b, a + b),
        Commands::Mul { a, b } => println!("{} × {} = {}", a, b, a * b),
        Commands::Greet { name, greeting } => println!("{greeting}, {name}!"),
    }
}

Running subcommands:

$ cargo run -- add 3.5 2.5
3.5 + 2.5 = 6

$ cargo run -- greet Alice --greeting "Hey"
Hey, Alice!

$ cargo run -- mul 6 7
6 × 7 = 42

For a full production CLI, combine clap with anyhow for error handling, indicatif for progress bars, and colored or owo-colors for terminal color output.

Wrapping Up

Concept	What it gives you
Threads	OS-level parallelism, compile-time data race prevention
Channels	Safe message passing between threads (mpsc)
Send & Sync	Compiler-enforced thread-safety contracts
Mutex & RwLock	Shared mutable state without data races
Atomics	Lock-free counters and flags
clap	Production-grade CLI argument parsing

The full series so far

Part	Topics
Part 1	Ownership, Borrowing, Lifetimes, Traits, Result/Option, Pattern Matching
Part 2	Closures, Iterators, Generics, Enums, Smart Pointers, Async/Await
Part 3	Macros, Modules, Cargo, Testing, Unsafe, FFI
Part 4	Threads, Channels, Send/Sync, Mutex, Atomics, clap

What's in Part 5?

Trait objects and dynamic dispatch (dyn Trait)
The Deref and Drop traits
Custom iterators
Building a real REST API with axum

Found this helpful? Drop a ❤️ and follow for Part 5!

Node Turns Waiting Into Events. Go Moves Context Switching Into User Space.

Tue, 05 May 2026 19:04:54 +0000

Most discussions of TypeScript/Node vs Go concurrency stop at the surface: Node is async, Go is threaded. That framing isn't wrong — it just isn't deep enough to be useful when you're picking a runtime, debugging a tail-latency problem, or explaining to your team why one of the services keeps falling over under CPU load.

The real difference is not async vs threaded. It's a question about where, in the system, suspended work lives — and what shape it takes when it's resumed.

tl;dr — Both Node and Go refuse to let the CPU sit idle while a request waits on I/O. They disagree on the unit of scheduling. Node's unit is the continuation — the tail of an async function captured as a heap closure. Go's unit is the goroutine — a full call stack the runtime can suspend and resume in user space. That single decision cascades into every other property of each runtime.

The Wrong Question

"Async vs threaded" is the wrong frame because it makes you think the choice is between paradigms. It isn't. Both runtimes have already made the same fundamental decision: do not block an OS thread waiting for slow external work. The interesting choice is how they implement that.

The actually useful question is:

When a request is waiting for I/O — for a database, an HTTP call, a Redis round-trip, a file read — what does the CPU do, and where does the suspended state of that request live?

Once you frame it that way, Node and Go aren't opposites. They're two answers to the same question — and each answer cascades into a different language shape, a different library style, and a different failure mode under load.

The naive blocking model answers the question with "an OS thread waits for the syscall to return." That model collapses around a few thousand concurrent connections — memory per thread, scheduler overhead, kernel context-switch cost. By 40,000 connections you're out of RAM, not CPU. Node and Go both refuse to do this. They diverge on which resource gets freed up and how the suspended work is captured for later resumption.

Node's Answer: Turn Waiting Into an Event

Node's model can be summarized in one line: the JS main thread only executes code that's already ready to run.

Look at this:

const user = await db.getUser(id);
return user;

It reads as if the function is paused, blocking on the database. It isn't. Here's what V8 actually does at the bytecode level when it compiles an async function: it rewrites the body into a state machine, with each await becoming a state transition.

The function above gets transformed into something equivalent to:

function asyncFn() {
  const promise = new Promise((resolve) => {
    let state = 0;
    const closure = {};                  // heap object holding locals

    function step(value) {
      switch (state) {
        case 0:
          state = 1;
          db.getUser(id).then(step);     // await → register continuation
          return;                         // ← function POPS here
        case 1:
          closure.user = value;           // resume: locals live in closure
          resolve(closure.user);
          return;
      }
    }
    step();
  });
  return promise;
}

Three things to notice:

await is not a pause. It's the point at which V8 returns from the function and pops the JS stack frame. The "rest of the function" is captured as a continuation registered on the awaited Promise via .then.
Local variables move to the heap. Because the stack frame is gone, locals (user here) live in a heap closure, accessible only when the state machine resumes.
Each await slices the function into another state. A function with two awaits runs in three event-loop turns, with three independently-pushed JS frames, with all live state stored in heap closures between them.

That third point is the most non-obvious. A single async function is not one unit of execution — it's a sequence of fresh frames separated by event-loop turns:

There is no "paused" function. There are only captured continuations and fresh frames that resume them. The event loop is the dispatcher: it watches for I/O readiness via libuv, for resolved Promises (via V8's microtask queue), for timers — and pulls the corresponding continuation onto the JS thread when it's ready to run. One thread can manage tens of thousands of concurrent connections, because at any moment only a handful of them have work that's actually ready.

This is event-driven concurrency in its precise sense — the runtime turns "waiting" into a registered event, and only resumes the captured continuation when the event fires.

The Visible Side Effect: Function Color

Because the suspension point has to be marked at compile time, async-ness becomes part of the function's type. A function that does I/O returns Promise<T>. Its callers must await it. Once they await, they themselves return Promise<T>. The "color" propagates up the call stack until you hit an async-aware entry point — typically the top of an HTTP handler or the event loop itself.

Bob Nystrom named this the function color problem in 2015. It's not a notation choice — it's a logical consequence of the stackless coroutine model. V8 cannot save and restore arbitrary JS call stacks. The only way to express suspension is "return a Promise and be marked async," and once one function does that, every function on the way up has to do the same.

The Hard Limit

The model fails the moment your code stops waiting. A single CPU-bound operation:

while (true) { /* heavy work */ }

…holds the JS main thread, and every other request on this process is dead until it returns. The event loop has nowhere else to go. Worker threads, child processes, or splitting CPU work into a separate service are real fixes, but they're escape hatches — they exist because the core model has only one main thread executing JS, and there is exactly one of it.

Go's Answer: Move Context Switching Into User Space

Go writes synchronous code:

user := db.GetUser(id)
sendResponse(user)

There is no await. There is no callback. The function looks like it blocks on the database. And yet the program scales to hundreds of thousands of concurrent operations on modest hardware.

The trick is that the scheduling boundary has been moved. Where Node has the programmer mark the suspension point with await and the runtime captures a continuation, Go lets the programmer write straight-line code and has the runtime suspend the entire goroutine when it hits a blocking I/O call.

This is the central insight, and the cleanest one-line statement of Go's concurrency model:

Go's essence is the user-space-ification of context switching.

A goroutine isn't an OS thread. It's a small (initially 2 KB) growable stack and a register snapshot, managed by the Go runtime. The runtime maps a large number of goroutines (G) onto a small number of OS threads (M) using scheduling contexts (P). This is the GMP model:

G — a goroutine. The unit of scheduling. Cheap to create, cheap to suspend.
M — an OS thread. Usually only GOMAXPROCS of them.
P — a scheduling context. Decides which G runs on which M.

many G  →  Go scheduler  →  few M  →  CPU cores

When a goroutine hits a blocking syscall or a channel wait, the Go runtime suspends the goroutine — saves its stack and registers — detaches it from the current M, and schedules another runnable goroutine onto that M. When the original goroutine's wait completes, it's marked runnable again, and some M eventually picks it up and resumes execution from the suspension point. None of this enters the kernel. No clone(2), no kernel-mediated thread switch, no kernel scheduler queue. The bookkeeping is all in user space.

That's the user-space-ification. The CPU still has to switch contexts when work shifts between goroutines, but the cost is roughly a function call plus a stack swap — not a kernel-mediated thread switch.

The key contrast with Node's model is in where the suspended state lives:

In Node, the JS call stack is shared and almost always near-empty — every async function in flight has already popped, with its state sitting in a heap closure. In Go, every goroutine owns its full call chain on its own heap-allocated stack; suspended goroutines look like frozen frames waiting for the runtime to resume them on some OS thread.

This is also why neither language can simply borrow the other's model. Node runs on V8, which was designed in 2008 for browser JS — single call stack, synchronous semantics, no concept of saving stacks across yields. Adding stackful coroutines would mean rewriting the engine, which is roughly what Java's Project Loom did to the JVM at huge cost. Go was designed from scratch with a runtime that owns stacks, can grow them, and can save them. The choice is locked in by runtime architecture, not language taste.

What "User-Space" Actually Buys You

The slogan only matters if user-space context switching is meaningfully cheaper than the kernel-mediated kind. It is — by more than an order of magnitude.

Two goroutines pinned to one OS thread (GOMAXPROCS=1), ping-ponging via runtime.Gosched() and via an unbuffered channel. Two pthreads pinned to one core (taskset -c 0), ping-ponging via pthread_mutex + pthread_cond. (Reproduction code at the end of the post.)

Measured on Intel N100, Ubuntu 24.04 (kernel 6.8.0), Go 1.23.4, gcc 13.3:

Operation	ns / switch
Goroutine yield (`runtime.Gosched`, GOMAXPROCS=1)	~102 ns
Goroutine round-trip via unbuffered channel	~436 ns (≈218 ns per G-switch + channel coordination)
pthread switch (mutex+cond ping-pong, single core)	~2,900 ns (range 2,818–3,611 across 5 runs of 2M iterations)

Ratio: roughly 28× cheaper for the bare scheduler yield, ~13× cheaper for the apples-to-apples synchronized round-trip.

Where the gap comes from:

Mode switch. The user → kernel → user round-trip alone is ~100 ns of entry/exit and ABI-mandated register save/restore. A goroutine switch never crosses that line.
Scheduler work in kernel space. Linux CFS maintains a red-black tree of runnable threads with locked, cross-CPU runqueues. The Go scheduler does the same job in user space with per-P local runqueues and lock-free fast paths — and skips the kernel locks entirely.
Cache and TLB effects. A kernel scheduler may migrate a thread to a different core, costing you cold L1/L2 and an instruction-cache reload. Goroutines normally stay on the same M, so the cache stays warm.

What the model does not buy you: a goroutine that makes a real blocking syscall still pays for a real OS thread switch — the runtime detaches the G from its M and may spin up another M so the rest of the goroutines keep running. Async preemption (Go 1.14+, signal-based) is the runtime's answer to tight loops that never yield, and it has its own cost. Once you saturate GOMAXPROCS, the user-space runqueue itself starts to show up in profiles.

The "user-space-ification" buys you cheap G-to-G switching on a hot M. That's where the order-of-magnitude lives. The syscalls, the M-to-M handoffs, the actual kernel work — those are still as expensive as they always were. The model wins by making the common case — many concurrent goroutines, mostly waiting, occasionally running — almost free.

(N100 is a low-power Alder Lake-N E-core; absolute numbers will be smaller on a server-class Xeon or EPYC, but the ratio is expected to hold.)

The Unit of Scheduling

The cleanest comparison is to ask what each runtime actually schedules:

	Node / TypeScript	Go
Unit of scheduling	callback / Promise continuation	goroutine
What's captured at suspension	tail of an async function as a heap closure	full call stack + registers
How code looks	explicit `async`/`await`	straight-line synchronous
Suspension marked by	the programmer (`await`)	the runtime (any blocking op)
Suspended state lives in	V8 microtask queue + heap closure	goroutine stack on the user-space heap
Kernel involvement	epoll/kqueue/IOCP via libuv	epoll/kqueue/IOCP via netpoller
CPU parallelism	one main JS thread; needs workers/cluster for cores	M:N scheduler runs goroutines across cores natively
Function color	yes (Promise infects up the call stack)	no (any function may block)
What breaks under CPU load	the entire event loop	nothing — scheduler runs another G on another M

The two columns describe deeply different mental models, but they belong to the same family. They are both user-space concurrency runtimes that avoid kernel thread-per-request. They differ in where the suspension is captured (the language vs. the call stack) and how broad the scheduler's mandate is.

Where the Boundaries Diverge: CPU-Bound Work

Node and Go look interchangeable on I/O-bound workloads. They diverge sharply the moment CP

The Concurrency Choreographer: Making BLE’s Async Data Sync with OBS

Tue, 25 Nov 2025 03:14:25 +0000

“What can we do with asynchronous programming?”, This was one of the questions I received in my talk on asynchronous programming that I delivered in Pycon MY. While I briefly mentioned building a chatbot that day. I believe the project we are discussing today should resonate more. During my spare time (however rare it is these days), I stream my gameplay session to document my journey. One of the widgets in my streaming session is a heart rate monitor, and it has been broken for some time.

Cute cartoon on the topic generated by Gemini

Seeking the Open Standard: From Fitbit’s Lock-in to Garmin’s BLE

I was doing some spring-cleaning for my OBS setup, and one component caught my eye — the heart rate monitor. For a time, it was a novel way to convey a streamer’s psychological state in a numerical form. Coincidentally, I was already wearing a fitness tracker: Fitbit Versa 3 and I decided to give it a go. Fortunately, there was a way to capture heart rate through the somewhat limited proprietary API, and have it streamed to my computer. Problem is, the fitness (not so) smartwatch died.

Funnily enough, a family member bought an almost identical Fitbit Sense a few months earlier and her watch is still functioning as of now. In a way the watch died rather prematurely. On the other hand, I was out of a job, and yet that didn’t translate to more gaming time as I was juggling with multiple things at the time. Consequently, the replacement only came much later, not long after I started my current job. Instead of getting another Fitbit, I moved on to a Garmin Instinct II, a tracker my peers highly recommended.

Unlike my past Fitbit devices, the Garmin watch does broadcast my heart rate via a standard Bluetooth interface. This makes replacing the workflow a much easier job. For that to work, I just needed to figure out how to get the data, and then supply that data to OBS via the WebSocket interface. The library in use is bleak to communicate with the watch via Bluetooth LE, and obsws-python to publish heart rate to OBS.

Previously, our chatbot project only dealt with asynchronous libraries. Yet, this time the library that communicates with OBS is synchronous. Therefore, we are taking this opportunity to discuss how to coordinate them to work with each other. Specifically, we are solving the problem where the producer (the heart rate monitor data) is asynchronous, and the consumer (the sending of heart rate to OBS) is synchronous.

Building the Async Producer: Connecting via Bluetooth Low Energy

My journey on tracking my fitness data started with reading about running. One of the books I read was titled Body, Mind and Sport by John Douillard. Some of the practices advocated may seem controversial, though the idea of being mindful about my physical state stuck in my mind. As a result, I started running with deep breaths, and soon noticed running could be made bearable or even enjoyable with some effort. At the same time, I went looking for a chest belt to pair with my fitness tracker installed on my phone.

That was before wearable tech went mainstream, so my choices were rather limited. Chest straps to measure heart rate were common among runners, but they weren’t built to work with Bluetooth. I had to search high and low to finally find one that would work with my smartphone application. It was slightly more expensive than an ordinary tracker, but it was worth it. I ran with it for a few years, and benefited from having my current heart rate displayed on my tracker in real time.

No more estimating by poking my fingers to my neck and doing multiplication while I was busy catching my breath.

Photo by Luiz Rogério Nunes on Unsplash

When wearable technology started to take off, Sony later released their smart band. The companion app also provided some limited insight on how my body reacts to activities in my day-to-day life. Conveniently, it also broadcasts my heart rate through Bluetooth and switching to it in my sports tracker took almost no effort. No more having to prepare my chest strap by wetting it before a run — that was a definite win.

Enough storytelling for now, let’s shift our focus back to the project. Like my old Sony smart band I bought second-hand, my Garmin Instinct 2 also broadcasts my heart rate on Bluetooth on demand. A quick search yielded blehrm, and it provides an abstraction to bleak. Unfortunately, it wasn’t the right library for the purpose, and I soon moved on to bleak directly after some failed attempts.

Firstly, we start by finding our device. Call the BleakScanner.discover method with a reasonable timeout, as shown below:

from bleak import BleakScanner

ble_devices = await BleakScanner.discover(timeout=10.0)
print(f"--- Discovered {len(ble_devices)} BLE Devices ---")
for i, device in enumerate(ble_devices):
    print(
        f"#{i}: Name: {device.name}, Address: {device.address}, Details: {device}"
    )

Running that will return (device addresses are replaced with the placeholder

--- Discovered 8 BLE Devices ---
#0: Name: Instinct 2, Address: <address>, Details: <address>: Instinct 2
#1: Name: None, Address: <address>, Details: <address>: None
#2: Name: None, Address: <address>, Details: <address>: None
#3: Name: U-WTD838C, Address: <address>, Details: <address>: U-WTD838C
#4: Name: None, Address: <address>, Details: <address>: None
#5: Name: None, Address: <address>, Details: <address>: None
#6: Name: None, Address: <address>, Details: <address>: None
#7: Name: None, Address: <address>, Details: <address>: None

Jot down the address, as we will need it for the next step: establishing a connection to the device.

from bleak import BleakClient

HR_MEASUREMENT_CHARACTERISTIC_UUID = "00002a37-0000-1000-8000-00805f9b34fb"
async with BleakClient(
    <address>,
    timeout=15.0
) as client:
    await client.start_notify(
        HR_MEASUREMENT_CHARACTERISTIC_UUID,
        heart_rate_handler,
    )

In the snippet, we are sending the address we collected earlier to BleakClient with a reasonable timeout. Then we subscribe to the heart rate measurement event, and delegate the handling to a function.

With the help of Gemini, we saved time digging through the documentation, as it returned a ready-made snippet for the BLE connection and data parsing, which is shown below.

def heart_rate_handler(sender, data):
    """Callback to handle heart rate measurement data."""
    # The heart rate data structure is defined in the BLE spec:
    # Byte 0: Flags (contains format of HR value and if other fields are present)
    # Byte 1: Heart Rate Value (8-bit or 16-bit)

    # Check the first bit of the flags to see if HR is 16-bit (0x01) or 8-bit (0x00)
    flags = data[0]
    is_16_bit = flags & 0x01
    if is_16_bit:
        # 16-bit HR value is at index 1 and 2
        hr_value = unpack("<H", data[1:3])[0]
    else:
        # 8-bit HR value is at index 1
        hr_value = data[1]
    logger.info(f"[{flags:04b}] Heart Rate: {hr_value} BPM")

There we have it, we have half the program done, and it is able to fetch heart rate data periodically from the device.

Bridging the Gap: The Producer-Consumer Pattern

Photo by Lenzil Gonsalves on Unsplash

Continuing the story after I bought my second-hand Sony smart band. Electronics products die inevitably, as Sony wasn’t making a viable successor for the product I had to look elsewhere. That was when I first looked at Fitbit, particularly a Fitbit Charge 3. It was more than doubled the price of my smart band, though it came with more features. One feature I particularly liked was the sleep tracker, as I have really bad sleep pattern. It was working fine, though with a major drawback — I lost the ability to read my heart rate through the sports tracker app.

Since the device came with a screen itself, it did mean there was no need to whip out my phone constantly to check my heart rate. Not being able to log my run with heart rate data attached created a minor annoyance. On the other hand, I moved on to other sports already, namely boxing and weightlifting. The shift in activity did change the need as I didn’t need the data as often. For instance, the short-burst nature of weightlifting did not require constant monitoring of my heart rate; the main focus was just on whether I could lift the dumbbell or otherwise.

Again, the tracker died after serving for several years. I moved on to a Versa 3 on a good deal, despite the mild annoyance. Other brands were considered, but I got confused with the numerous lines of products and decided to stay with Fitbit for another time. Being a smart watch, Fitbit did release an SDK for their Versa watches, but they discontinued application development support when they released the successor, Versa 4. Regrettably, I didn’t put in the time and effort to develop anything as I was adequately happy with it.

As mentioned earlier, I found a way to broadcast my heart rate with an app offered in the Fitbit app store, and that led to the setup of a heart rate widget on my OBS setup. Coincidentally, I acquired Tetris Effect on my workstation, so I was curious to see if my heart rate changed throughout the game play. Surprisingly, it really did, and the recording clearly showed the heart rate fluctuations in action.

https://medium.com/media/018de8b0f30246afd93165f500a77b62/href</a></iframe>

Granted, as the tracker was worn on the wrist, the response was somewhat delayed. Despite that, the stream showed as the tetrimino dropped faster, my heart rate also increased correspondingly. More often than not, the height of dropped blocks waiting to be eliminated also played a role in influencing the heart rate.

Now that we have the stream of heart rate data, how do we recreate the widget in OBS through our script?

Open Broadcasting Studio, or OBS in short, offers a way to interact with the sources on screen through the websocket interface. In our case, we are interested in sending our heart rate data to a text source. By using the obsws-python library, we can send the data as soon as we capture it from the device. We can establish this connection to OBS using the following simple code:

from obsws_python import ReqClient

obs = ReqClient(
 host=<host>,
 port=<port>,
 password=<password>,
)

All of these settings can be found in the OBS websocket setup page, as shown in the screenshot below:

Sending heart rate to the source is simple, through the obs.set_input_settings method

obs.set_input_settings(
 <identifier of the target source>,
 {“text”: f”{item:3d}”},
 True,
)

By assembling all these together we would have a complete program for the job. The only problem with putting everything together is that we are mixing asynchronous code calling bleak with a strictly synchronous and likely non thread-safe code by obsws-python. Fortunately, the data only flows one-way; otherwise it complicates the whole setup significantly more. Ideally the library that dealt with OBS websocket could be written in a more asynchronous manner but obsws-python was the only reliable library that I could find in a weekend.

The project started as a weekend project, and I stopped right after achieving a running proof-of-concept. Only later, when I finally had the time, did I start refactoring this into a more reliable program that properly handles errors and graceful shutdown. Considering we discussed rather extensively in the past on those topics, let’s skip to how we turn this into a producer-consumer design.

with ThreadPoolExecutor(max_workers=5) as executor:
   executor.submit(producer)
   executor.submit(consumer)

We want both the consumer and producer to remain isolated from each other, so we separate them into individual threads. Usually it is better to split them into different processes, but since we are only running one event loop for the producer part it is okay to stick with threads. In order to synchronize both of them, we need to pass a queue and an exit event to them as well. This can be achieved with functools.partial, or by writing a class with the dunder __call__ method, a topic we discussed previously. I will leave the implementation detail out to focus on the core concept itself.

Now, back to our producer, where we collect the heart rate, instead of printing the heart rate, we dump it to our queue:

logger.info(f”[{flags:04b}] Heart Rate: {hr_value} BPM”)
asyncio.create_task(asyncio.to_thread(queue.put, hr_value))

In case you missed our series of articles on AsyncIO (go read them, they are awesome!), queue.put is a blocking operation, so we are delegating it to asyncio.to_thread. For our purpose, we don’t exactly care about the outcome, but we want to ensure it is scheduled safely. Therefore, we pass the returned coroutine object to asyncio.create_task.

On the other hand, as the code dealing with OBS is synchronous, we need to plan carefully.

def consumer(queue: Queue, exit_event: Event) -> None:
    obs = ReqClient(
        host=<host>,
        port=<port>,
        password=<password>,
    )

    logger.info("Starting consumption loop")
    while True:
        # task 1: handle the exit event
        if exit_event.is_set():
            logger.info("Exiting consumption loop")
            break

        # task 2: publish to obs
        with suppress(Empty):
            if item := queue.get(timeout=5.0):
                obs.set_input_settings(
                    <identifier of the target source>,
                    {"text": f"{item:3d}"},
                    True,
                )

This simplified snippet sets up an infinite loop that does two things, firstly it checks if the exit_event is triggered, and secondly, it consumes the heart rate data from the queue and sends it to OBS. We would need to set a timeout and suppress the potential Empty exception when getting a value from the queue. The reason for this is otherwise the code will block until it receives a data, and we would never be able to capture exit_event that was emitted elsewhere.

Asyncio’s Role in Real-World, Non-Web I/O

Essentially, this is a fairly common Consumer-Producer application between two different components. While the consumer is depending on the producer to publish to OBS, the producer should not have to wait for it to complete. By separating them into different threads and using a queue for synchronization, we allowed both components to work independently. Unlike the telegram chatbot we built previously, this shows how we can adapt synchronous code into an asynchronous environment.

I appreciate the fact that Garmin allows the heart rate data to be made available through an open interface. If I were to get a new Versa watch, this project would not have been possible as the data is locked away from a proprietary API. As mentioned earlier, Fitbit also removed support for application development for the newer models.

That would mean no more Tetris Effect streams with heart rate readings for me.

https://medium.com/media/6fd1166a20e17dcd450a99014354e282/href</a></iframe>

Not like anyone cared anyway.

So, is this project a good example as a response to the question posed early in the article? Is this a good demonstration of what asynchronous programming is capable of? I would argue yes, as this shows even though not all components are asynchronous, it is still possible to workaround it. When we have multiple components that need to talk to each other, asyncio is definitely a way to go, albeit requiring more thought and work into it.

That’s all I have for today! The code snippets are purposefully simplified for brevity, but if you are interested in the project, feel free to check it out on GitHub. Thanks for reading, I shall see you in writing again soon.

For the sake of transparency, this article was a collaborative effort with my editor, Gemini, who assisted me throughout the writing process. Beyond providing editorial suggestions to tighten the narrative and structure, the model was instrumental in accelerating the technical side of the project, specifically by generating and verifying the initial code snippet required to correctly extract and parse the raw heart rate data from the BLE characteristic — a small but critical step that saved me hours of digging through specification sheets. The final code and the story are mine.

Concurrency vs. Parallelism: Achieving Scalability with ProcessPoolExecutor

Tue, 08 Apr 2025 14:57:43 +0000

In our previous exploration, we dove deep into AsyncIO by writing a task scheduler. We discovered how IO-bound operations were scheduled to the event loop concurrently while building our task scheduler project. Building on what we’ve done, we will revisit the project and scale it to run CPU-bound operations.

More chefs, more dishes, illustration generated by Microsoft Copilot

How to write an AsyncIO Telegram bot in Python
Understanding Awaitables: Coroutines, Tasks, and Futures in Python
AsyncIO Task Management: A Hands-On Scheduler Project
Concurrency vs. Parallelism: Achieving Scalability with ProcessPoolExecutor

AsyncIO: A Foundation for Concurrency

Photo by John Barkiple on Unsplash

It is hard to avoid talking about concurrency in discussions about asynchronous programming. However, when it gets mentioned, often times it gets mixed up with parallelism. When Golang came out, people raved about how goroutines are efficient, and that was the first time I hear people discussing concurrency after my undergrad days. Rob Pike eventually delivered a talk aiming to distinguish the two concepts and it became quite popular among my peers.

While building our project last week, I mentioned event loops functions like a timetable where we slot in awaitables for execution. They would be scheduled to run one at a time, switching to another whenever one is done, or is busy doing IO-operations. For example, when we send multiple dex tasks, it would attempt to (order is not necessarily exact, just to illustrate the idea)

Start an HTTP request to the PokeAPI
Wait for the response for (1)
Start another HTTP request to the PokeAPI
Wait for the response for (3)
Continue the task started in (1), and compute the result
Start another HTTP request to the PokeAPI
Wait for the response for (6)
Continue the task started in (3), and compute the result
Continue the task started in (6), and compute the result

We can see the event loop improves the efficiency by keeping the CPU busy, while IO is happening in the background. This is an example of concurrency, in Pike’s word: dealing with a lot of things at once. In a way, it is like planning our time to minimize time waste on waiting in the kitchen. For example, we can proceed to cut ingredients for other dishes, while waiting for our pot of soup to boil. This allows us to concurrently prepare or cook multiple dishes at once.

So we now know we can achieve concurrency with AsyncIO for IO-bound operations. How about CPU-bound operations?

We are fortunate to live in an era where computing devices sports multicore CPU by default. Theoretically this should improve efficiency, but spreading work to multiple cores is a challenge. Continuing our cooking analogy, having an extra core or CPU is like having another kitchen and chef. Both chefs can now work together and get more dishes prepared at once. They can both choose to schedule their time wisely, to work concurrently as well.

Go’s goroutines can be scheduled to both run concurrently, or to be distributed to multiple CPU cores to run at once. In Python, we can achieve concurrency with AsyncIO. However, for parallelism, we would have to use tools like ProcessPoolExecutor in addition to AsyncIO. Let’s get started by continuing our example project last week by making fib run in parallel.

Harnessing Parallelism: ProcessPoolExecutor in Action

Photo by Becca Tapert on Unsplash

We usually opt to spawn a new thread, when we want to distribute work across the available cores. With threads, data can be shared among the main process and the spawn threads. However, it also means a race condition can occur when multiple threads are writing to the same variable. Hence, the global interpreter lock (GIL) was introduced, to limit the behaviour. While it protects the shared data, it makes threads inefficient as it depends on the lock.

Ensuring data is properly shared across processes and threads is therefore one of the biggest challenge in parallelism. There are ongoing efforts to deal with the limitations, or even removing GIL entirely in recent releases. But for now, most of the time we would just start new processes to distribute works to be done at once. In our case, let’s set up ProcessPoolExecutor in our repl function:

from concurrent.futures import ProcessPoolExecutor

async def repl() -> None:
    with (
        ProcessPoolExecutor() as executor,
        suppress(asyncio.CancelledError),
    ):
        ...

Previously, in our evaluator, we would spawn a new thread whenever we work on fib, through the use of asyncio.to_thread. We also know it is just an abstraction of loop.run_in_executor. Now that we are using our own process executor, let’s revert to use run_in_executor, with our own ProcessPoolExecutor

async def evaluate(…):
    match expression:
        ...

        case Expression(command=Command.FIB):
            return asyncio.get_running_loop().run_in_executor(
                executor, fib, expression.args[0]
            )

Now, whenever we issue a fib command, a new process is spawn instead of a thread to execute the (potentially) super slow CPU-bound operation. Starting a new process is considered more expensive, as we are pending resource allocation from the operating system. Sharing data and synchronization between processes also now require more effort, and we will discuss further about this later.

The loop.run_in_executor is essentially an abstraction of executor.submit call, where it takes function and the corresponding parameters. Therefore, in general, we would write it as

with ProcessPoolExecutor() as executor:
    future = executor.submit(fib, 100)

The returned future object, has a nearly identical design compared to the asyncio counterpart, sans the asynchronous magic.

While we are at it, let’s add a new command to send multiple fib operations, where the format is fib-multi [ …]. For instance, when we do fib-multi 100 200 300, it would schedule fib 100, fib 200 and fib 300 to run in parallel. For that, we add a new case in our evaluator

 async def evaluate(...):
    match expression:
        ...

        case Expression(command=Command.FIB_MULTI):
            return asyncio.gather(
                *[
                    asyncio.get_running_loop().run_in_executor(
                        executor, fib2, nth
                    )
                    for nth in expression.args
                ]
            )

When we use asyncio.gather, it takes all the awaitables passed as parameters and return a Future object. The result of the awaitables will be returned as a list when .result() is called. In our case, since we are setting the executor explicitly to a ProcessPoolExecutor, the futures are then scheduled to run in parallel.

And for that, we need to revise our job and dash to properly display the result, namely to handle cases where the result returned is a list.

def job(
    task: tuple[
        str,
        Event | None,
        asyncio.Task[str] | asyncio.Future[str] | asyncio.Future[list[str]],
    ],
) -> str:
    try:
        match task[-1]:
            case asyncio.Future():
                return (
                    "; ".join(task[-1].result())
                    if isinstance(task[-1].result(), list)
                    else task[-1].result()
                ) # type: ignore

            case _:
                return "Still waiting"

    except asyncio.InvalidStateError:
        return "Still waiting"

def dash(
    tasks: Sequence[
        tuple[
            str,
            Event | None,
            asyncio.Task[str] | asyncio.Future[str] | asyncio.Future[list[str]],
        ]
    ],
) -> str:
    result = [
        "\t".join(("id", f"{'command':16s}", "done?", "result")),
        "\t".join(("==", f"{'=' * 16}", "=====", "======")),
    ]

    for num, (line, _, task) in enumerate(tasks, 1):
        task_done, task_result = False, None

        if task_done := task.done():
            task_result = (
                "; ".join(task.result())
                if isinstance(task.result(), list)
                else task.result()
            )

        result.append(
            "\t".join(
                (
                    str(num),
                    f"{line:16s}",
                    str(task_done),
                    task_done and str(task_result)[:16] or "",
                )
            )
        )

    return "\n".join(result)

We can now issue an actual fib-multi command, and check our system monitor to see the operating system does indeed allocate multiple processes to do the job.

A screen-capture from my system monitor

On the other hand, dex is still unaffected, and would continue to work concurrently despite the changes introduced to fib. This exercise show how it is possible to introduce parallelism into a concurrent program.

Managing Processes: Cancellation and Communication

Photo by Mark Boss on Unsplash

A big drawback of starting a new process is that communication takes a lot of effort, and we lose some control over it. For instance, if we create a task in asyncio, we can cancel the task whenever needed with .cancel(). However, despite the ProcessPoolExecutor provides a .cancel() method as well, it does not always work for scheduled futures that are in progress.

One way to workaround the problem is through event objects. In our example, we will need 2 types of events, one for task cancellation, and another for application exit. Let’s get fib to subscribe to both of them.

def fib(nth, exit_event=None, cancel_event=None):
    ...

    for i in range(1, nth + 1):
        if (exit_event and exit_event.is_set()) or (
            cancel_event and cancel_event.is_set()
        ):
            return "Cancelled"

        ...

The key is to stop the loop, whenever either of the events is set. In our case, would be either user is exiting the program with quit, or cancelling the job with kill . Now we move on to the evaluator to ensure both these events are sent to fib.

from multiprocessing.managers import SyncManager

async def evaluate(
    …
    executor: ProcessPoolExecutor,
    exit_event: Event,
    manager: SyncManager,
    expression: Expression,
):
    match expression:
        ...

        case Expression(command=Command.FIB):
            cancel_event = manager.Event()
            return cancel_event, asyncio.get_running_loop().run_in_executor(
                executor, fib, expression.args[0], exit_event, cancel_event
        )

        case Expression(command=Command.FIB_MULTI):
            cancel_event = manager.Event()
            return cancel_event, asyncio.gather(
                *[
                    asyncio.get_running_loop().run_in_executor(
                        executor, fib, nth, exit_event, cancel_event
                    )
                    for nth in expression.args
                ]
            )

The application exit_event is going to be applicable for all fib operations, so we are requiring it in the parameters. On the other hand, a cancel_event is only relevant for a particular job, hence it is created just for it. Then we set up the global exit_event in repl().

import multiprocessing

async def repl() -> None:
    with (
        ProcessPoolExecutor(max_workers=5) as executor,
        suppress(asyncio.CancelledError),
    ):
        manager = multiprocessing.Manager()
        exit_event = manager.Event()
        …

We create Event() object through the multiprocessing.Manager() object because it is generally more reliable, especially we are passing the object between processes.

The ShutdownHandler would require a little bit of revision so it is aware of the exit_event.

@dataclass
class ShutdownHandler:
    exit_event: Event
    signal: int | None = None

    async def __call__ (self) -> None:
        logger.info("Shutting down tasks. signal:%s", self.signal or "")

        self.exit_event.set()

        ...

Now we can prompt fib, which is our CPU-bound task to return when the application exits. The changes in the handler, also require init() function to take exit_event as parameter

def init(exit_event: Event) -> None:
    loop = asyncio.get_running_loop()

    for sig in (signal.SIGTERM, signal.SIGHUP, signal.SIGINT):
        handler = ShutdownHandler(exit_event, sig)
        loop.add_signal_handler(sig, lambda: asyncio.create_task(handler()))

    loop.set_exception_handler(ExceptionHandler(ShutdownHandler(exit_event)))

Now the application exit part is done, let’s move on to task cancellation. In the evaluator, we see the CPU-bound jobs also return the cancel_event, so we need to properly store them in our list of tasks in repl()

async def repl() -> None:
    ...

                match result := await thread_last(
                    line,
                    (tokenize, t),
                    (parse, g),
                    (
                        evaluate,
                        client,
                        executor,
                        exit_event,
                        manager,
                        tasks,
                        ShutdownHandler(exit_event),
                    ),
                ):
                    case (_, asyncio.Future()):
                        tasks.append((line, *result))
                        print(f"Task {len(tasks)} is submitted")

                    case asyncio.Task():
                        tasks.append((line, None, result))
                        print(f"Task {len(tasks)} is submitted")

                    case _:
                        print(result)

And the corresponding kill function is now

def kill(task):
    match task[-1]:
        case asyncio.Task():
            task[-1].cancel()

        case asyncio.Future():
            assert task[1]

            task[1].set()

    return f'Task "{task[0]}" is killed'

Note that asyncio.Task is a subclass of asyncio.Future, so the case order matters.

The Power of the Hybrid Approach

Photo by Elena Mozhvilo on Unsplash

In our example, we can use ProcessPoolExecutor to introduce parallelism to our application. This allows us to scale CPU-bound operations so that they can be distributed to other cores of the CPU, allowing them to be executed simultaneously. On the other hand, IO-bound operations are still scheduled to run concurrently.

Remember the cooking analogy we mentioned earlier? Doing things concurrently is very similar to cooking. It is possible to prepare multiple dishes at a time with clever time management. While we wait for our chicken in the oven, we can go stir-fry whatever that is in the pot.

On the other hand, doing it in parallel means adding a new chef (and maybe even another kitchen). Adding a new chef, would definitely increase throughput, allowing more dishes to be done at once. On the other hand, either or both of them can still be working concurrently, increasing efficiency. In our case, it would be starting separate processes, and each manages their own event loop to take on tasks.

I hope I am doing a good enough job in this tour of AsyncIO and asynchronous programming. I certainly learned a lot throughout these few weeks, revisiting the documentation, and through building the example project. The updated project can still be accessed on GitHub ,,) with week1 and week2 code separated, allowing comparison. Thank you again for your time, and I shall write again, next week.

This article was enhanced by the editorial assistance of a large language model, which provided valuable feedback on clarity, flow, and adherence to writing principles. While the core ideas and code remain my own, the LLM’s suggestions significantly contributed to the overall quality and readability of the piece. I am always open to collaboration on interesting projects and exploring job opportunities. Feel free to connect with me here on Medium or via my LinkedIn profile to discuss further.

Coder Legion - Recent posts tagged concurrency

How to Lock a Function Across Processes with a Decorator in Python

The Decorator

How to Use It

How It All Works

A Shared Lock File

Wrapping the Original Function

Attempting to Acquire the Lock

Pausing and Executing

Releasing the Lock

The Limits of This Approach

Summary

Why Local LLM Inference Needs Admission Control Before It Needs More Hardware

Profiling a Go Service in Production: pprof in 10 Minutes

Enabling pprof

Collecting a CPU profile

Reading a flame graph

Problem 1: Excessive allocations

Problem 2: Goroutine leak

Problem 3: Lock contention

A quick reference for which profile to use

One workflow for production incidents

Summary

context.Context Is Not Optional: A Practical Guide to Cancellation in Go Services

What context.Context actually does

What happens without context

The correct structure: context flows in, errors flow out

Wiring context to OS signals for clean shutdown

Timeouts: per-job vs. pool-wide

Checking context in CPU-bound work

The three rules

Summary

Why Modern Engineering Teams Must Prioritize System Observability Over Raw Performance

Building a Concurrency-Safe Movie Reservation System

Building a Concurrency-Safe Movie Reservation System with FastAPI, Redis, and PostgreSQL

Introduction

The Problem

Designing the Reservation Workflow

Layer 1: Distributed Seat Locking with Redis

Preventing Partial Seat Locking

Lock Ownership Verification

Database Constraints as the Final Safety Net

Transactional Reservation Processing

Idempotent Reservation Requests

Protecting the API with Rate Limiting

Background Workers

Stress Testing the System

What I Learned

Final Thoughts

Rust Concurrency: Threads, Channels, Mutex & Sync (Part 4)

series: Core Rust Concepts

Table of Contents

19. Threads

Spawning a thread

Moving data into a thread

Spawning many threads

20. Message Passing with Channels

Basic channel

Multiple producers

Synchronous channel (bounded)

21. Send and Sync Traits

22. Mutex and RwLock

Mutex

RwLock — many readers, one writer

Avoiding deadlocks

23. Atomics

Memory ordering cheat sheet

24. Building a CLI Tool with clap

Basic CLI

Subcommands

Wrapping Up

The full series so far

What's in Part 5?

Node Turns Waiting Into Events. Go Moves Context Switching Into User Space.

The Wrong Question

Node's Answer: Turn Waiting Into an Event

The Visible Side Effect: Function Color

The Hard Limit

Go's Answer: Move Context Switching Into User Space

What "User-Space" Actually Buys You