Coder Legion - Recent posts tagged latency

The 500ms Budget That Shapes Every Voice Agent You Build

Tue, 08 Sep 2026 02:00:32 +0000

Most of us here have built a chatbot at some point, and going from text to voice looks like it should just be two more API calls, one on each end. It is not, and the reason is a single latency number that reorganizes the whole pipeline.

A voice agent runs the same loop every turn: capture audio, transcribe it, run it through a model, generate a reply, synthesize speech, play it back. That entire round trip has to land under about 500 milliseconds or the caller hears a gap and assumes the line dropped. A text chatbot can sit there for two or three seconds and nobody blinks, so most of the habits you carry over from text work are suddenly too slow.

That budget is why production voice stacks look the way they do. Speech recognition runs in streaming mode so transcription starts before the caller stops talking, which lets the language model begin working on a partial sentence instead of waiting for a complete one. Modern ASR from Deepgram, AssemblyAI and Google sits under a 5 percent word error rate on clear English, so accuracy is largely a solved problem. The engineering work is all about where the milliseconds go.

The other thing that eats the budget is end of turn detection. Voice activity detection has to decide when you have actually stopped speaking, and that call happens before the model sees a single token. Cut it short and you clip people mid sentence, wait too long and your real time agent feels like a phone tree.

If you want the full pipeline breakdown, the open source options like LiveKit, Pipecat and Vocode, and where the cost actually lands, AI voice agents covers it end to end.

The cost comparison surprised me more than the latency did. A US call center seat runs $25 to $65 an hour once you count benefits, training and management overhead, while a voice agent handling the same calls costs $0.05 to $0.25 a minute with no ramp up and unlimited concurrency. That gap is why enterprise production deployments grew 340 percent year over year through early 2026.

Curious what people here have hit in practice. If you have shipped a voice agent, where did your latency actually go, the ASR, the model, or the TTS?

How I tried to write an article about slow Chinese LLMs

Thu, 06 Aug 2026 15:51:59 +0000

Recently, I've added a bunch of hype-monsters to my AI Werewolf:

Kimi K3
Qwen 3.8 Max, Qwen 3.7 Plus, Qwen 3.7 Flash
MiniMax M3

Plus the ones I've had for a while

DeepSeek V4 Pro and Flash
GLM-5.2
Sakana Fugu base and Ultra

The last one is Japanese, not Chinese, but it was in the news a month or two ago and it belongs in this story.

I sat down to write about how slow Chinese models are. Because they are all annoyingly slow even with a small context. I had the numbers, the thesis wrote itself, and then... I discovered something I didn't expect.

Okay, let me go straight to the problem

All Chinese official APIs are extremely slow. DeepSeek got better with their v4, the rest are just terrible. So slow that they are barely usable in a text game. No, seriously, take a look.

Time to produce one four-sentence vote:

Kimi K3: 29 to 34 seconds
MiniMax M3: 25 to 30 seconds
Qwen 3.8 Max: 25 to 27 seconds, and only because I capped its thinking. Uncapped it hit 100.
DeepSeek V4 Pro: 14 to 22 seconds, the best of the group

Same prompt, same afternoon: Claude 5 Opus answers in 5.9 seconds.

I have a test which simulates the voting at the end of a game day. Some chat has happened, some players have already cast their votes, and now the model under test has to do the same. The prompt contains all of that: 36,000 characters, which works out to 8-13k tokens depending on the model's tokenizer. Not much, assuming those models have 1M contexts.

More US/EU models:

Model	Time	Input	Output tokens	Avg cost
GPT-5.6 Luna	3.2-3.6s	12,124	190-231	$0.0020
Mistral Large 3	2.9-5.2s	8,175	73-134	$0.0042
Claude Fable 5	4.8-6.0s	13,027	183-209	$0.1399
Gemini 3.6 Flash	4.6-7.1s	7,979	297-532	$0.0152
GPT-5.6 Sol	6.7-7.3s	12,124	150-156	$0.0471
Claude 5 Opus	5.9-7.8s	13,027	220-387	$0.0726
Gemini 3.1 Pro	11.6-12.1s	7,979	821-940	$0.0263
Claude 5 Sonnet	6.8-14.8s	13,027	252-821	$0.0464
Grok 4.5	10.0-14.8s	8,190	428-607	$0.0149
Mistral Medium 3.5	2.9-35.0s	8,175	57-109	$0.0128

Worth staring at the input column for a second. The same 36,000 characters is 7,979 tokens to Gemini, 8,175 to Mistral, 12,124 to OpenAI and 13,027 to Claude. Claude is billing me for 63% more prompt than Google is, for the identical text, before anyone has thought about anything.

Frontier models sit in the 3-7 range. Grok is lagging behind. Mistral is weird - their Large is way faster than the newer Medium. Gemini Pro is also not great - Google, where are your TPUs?

Mistral Medium deserves its own note. Seven runs of the same prompt across one day: 3.3s, 3.6s, 24.1s, 35.0s, 27.9s, 11.1s, 2.9s. Output was 57 to 109 tokens every single time. That's their infrastructure having a day. Whichever single measurement I had taken, it would have been wrong about this model.

Now the other group.

Model	Time	Input	Output tokens	Avg cost
Qwen3.7 Flash *	8.4-10.5s	8,216	1,093-1,174	$0.0004
GLM-5.2	16.4-21.2s	8,150	1,111-1,406	$0.0138
DeepSeek V4 Pro	14.4-22.2s	8,164	773-1,392	$0.0009
Qwen3.8 Max *	25.4-27.3s	8,250	1,120-1,159	$0.0175
DeepSeek V4 Flash	13.3-30.1s	8,243	990-2,275	$0.0005
MiniMax M3	25.3-30.4s	8,037	1,216-2,381	$0.0045
Qwen3.7 Plus *	12.9-31.0s	8,216	693-1,158	$0.0038
Kimi K3	28.9-33.5s	8,192	744-1,103	$0.0310

* Qwen rows are already capped at 1,024 reasoning tokens. Uncapped numbers are further down, and they are worse.

The only one under 15 seconds needed a thinking cap to get there. The rest need 13 to 34 seconds to produce a four-sentence vote. Twelve bots take turns talking in my game. At 30 seconds a turn, one round of discussion is six minutes of watching a spinner.

That was going to be the article.

Except for one thing

Claude 4.5 Haiku was slow.

Model	Time	Input	Output tokens	Avg cost
Claude 4.5 Haiku	19.7-42.0s	8,906	1,769-4,095	$0.0231

Very slow. Why? Isn't it the smallest, the cheapest, and therefore the fastest model? Apparently, it's one of the slowest in the roster.

What is going on there?

The pipes are fine

My first guess was - the hardware. The US has better hardware, right? And its servers are closer - less network latency.

Well, not really. Divide output tokens by seconds and you get a rough throughput per call, prefill included, across every run I have of each:

Qwen3.7 Flash: 104-145 tokens/s
Claude 4.5 Haiku: 90-98 tokens/s
DeepSeek V4 Flash: 73-76 tokens/s
Gemini 3.1 Pro: 70-78 tokens/s
GLM-5.2: 57-74 tokens/s
MiniMax M3: 48-78 tokens/s
Claude 5 Opus: 37-49 tokens/s
Qwen3.8 Max: 42-44 tokens/s
Kimi K3: 26-35 tokens/s

Look at DeepSeek V4 Flash. Three runs, 990 then 1,415 then 2,275 output tokens, and the generation rate never moves off 73-76 tokens/s. The connection is steady. The model just decided to write twice as much the third time, and the wall clock went from 13.3s to 30.1s.

DeepSeek and MiniMax generate as fast as anything Google runs. They take five times longer because they write eight times more. Only Kimi and Qwen Max are also slow per token, and even for them the token count does most of the damage.

Funny enough, Haiku types faster than every model here except Qwen 3.7 Flash, and it is the slowest model I have. 90 tokens a second, 42 seconds to answer. It isn't struggling to produce text. It produces four thousand tokens of it to say four sentences.

So the whole table reduces to one number: how many tokens does the model decide to write before it answers a question that needs four sentences. Opus wrote 220 to 387. Haiku wrote 1,769 to 4,095.

They think differently

All the latest Anthropic models support adaptive reasoning: the model reads the request and decides for itself how much thinking it deserves. Haiku doesn't, it's too old. Opus 5 outputs 220 to 387 tokens on this prompt, and that includes the answer and the reasoning. GPTs are the same, Sol outputs even less (~150 tokens). Gemini 3 does it too, though it lands on a bigger number - Google calls it dynamic thinking, and the old thinking budget is deprecated on that generation. Grok doesn't do it at all, it runs at fixed high effort, and that's why it takes 10-15 seconds.

Every Chinese model in my roster reasons at close to full tilt on every request, whatever the request is:

Kimi K3 always runs at max effort. There is no supported way to turn it down. Roughly 85-90% of its output tokens are reasoning.
Qwen offers a thinking_budget you set up front. The default is 4096, and that default is what produced the 100-second runs below.
DeepSeek and GLM ship thinking-only entries with no depth control I can reach.
MiniMax M3 advertises adaptive thinking, and this is the interesting one. It has the feature by name and still spent 1,216 to 2,381 tokens deciding who to vote for.

Every one of those is defensible if you're building a coding agent or answering a hard question. A reasoning model is graded on benchmarks where thinking longer nearly always scores better and nothing charges it for the tokens. Left alone, thinking hard on everything is the behavior you get, and it's the behavior that wins the leaderboard I'm not running.

My game is a different animal. Waiting for 30 seconds for a bot to cast a vote is devastating. Imagine my frustration seeing this slowness in a basic chat, while the rest of the world is celebrating Chinese AI winning the race.

The lottery

Oh, and this performance is super inconsistent. Qwen 3.8 Max shows it best. Before I capped it, four runs of the identical prompt: 26.3s, 30.6s, 81.9s, 100.5s. The output tokens went along with it: 1,068, 1,149, 3,363, 4,169.

Line those up and the generation rate is 40.6, 37.5, 41.1 and 41.5 tokens per second. Close to constant. The model was working at the same speed the whole time. It just decided, twice, that this question deserved three times more thought.

Nothing changed between those runs. Same prompt, same day, same server, same everything. What moves is the chain of thought itself, and a chain of thought is a sequence the model writes one token at a time with a temperature attached. I run Qwen at 0.7. Early in the trace it either reaches for "let me re-read what Kenji said on day one" or it doesn't, and that one fork decides whether you get 1,100 tokens or 4,100.

So it isn't a spread around some typical value, it's closer to two outcomes. Max's four runs: 1,068, 1,149, then 3,363, 4,169. Two short, two long, nothing in between. Qwen Flash did the same - 1,831, 1,897, 1,912, and then one run at 3,035. Haiku went 1,769, 2,635, 4,095.

You can't provision for that. A model with a 27-second typical case and a 100-second bad case needs a timeout set for the bad case, and a player who has to sit through it.

Low thinking budget

Qwen's API takes a thinking_budget. I capped all three Qwen models at 1,024 reasoning tokens and Max settled into 25.4-27.3s. Worse reasoning, bearable latency. That is a trade I will take for a game where a human is sitting there waiting.

The cap only helps the models that were over it. Flash was writing 1,831 to 3,035 tokens uncapped, so the cap cut it to 8.4-10.5s. Plus was already choosing 760 to 1,262, mostly under the limit, so capping it changed nothing and it still turned in a 31-second run.

For comparison, Gemini 3.1 Pro writes about as much as capped Qwen Max, 821 to 940 tokens, and comes back in 11.6-12.1 seconds every single run. Similar output, less than half the wait, and none of the spread. Nobody set a budget on it either. It runs at high effort and picks that number itself.

MiniMax M3 has no budget parameter at all. Its adaptive setting is the only throttle, and the only stronger lever is turning thinking off entirely. Kimi K3 has an undocumented K2-era thinking: disabled toggle I don't want to build on. For those two, what you measure is what you get.

The money is the good news

The cheap ones are properly cheap:

Qwen3.7 Flash: $0.0004 a vote
DeepSeek V4 Flash: $0.0005
DeepSeek V4 Pro: $0.0009

DeepSeek V4 Flash wrote 990 to 2,275 output tokens against Mistral Large's 73 to 134, and it still cost eight times less.

Then there is Kimi K3. It bills $3 in and $15 out, the same rate card as Claude 5 Sonnet. Ignore the averages in the table for a moment, since one of K3's three runs caught a warm cache: on cold runs K3 cost $0.036 and $0.041, Sonnet $0.043 and $0.045. Near enough the same money for a reply that took four times longer, and 85-90% of K3's output tokens were reasoning that nobody ever sees. Note also that K3 is being billed for 8,192 input tokens where Sonnet is billed for 13,027 of the same text.

The Qwen cap helped here too. Max's runaway 4,169-token think cost $0.0415 against $0.0229-0.0235 for a normal run. The lottery is a billing lottery as well as a latency one, and capping the thinking capped both.

Caching works on all of them, which matters when you re-send a growing conversation every turn. Those wide cost ranges in the tables are this effect. I ran the sweep twice nine minutes apart, and on the second pass Kimi went from $0.0411 to $0.0163 and GLM from $0.0167 to $0.0083, with the same prompt and more output tokens than the first time.

MiniMax reported cache hits on my very first probe call, before I had done anything deliberate about it. Qwen's implicit cache cut a real call from $0.0229 to $0.0115 between two runs. DeepSeek reads cached input at 2% of the normal rate, the deepest discount in my whole roster.

And then there is Fugu

Sakana's Fugu Ultra is Japanese, and it is seven times slower than the slowest model in that second table.

It is so slow that my tests were timing out after 3 min of waiting for a response.

The bill for one successful vote was $0.585. About 65% of that was orchestration_input_tokens and orchestration_output_tokens, internal drafts and re-reads that never reach me. I don't know what it is orchestrating, but this is not usable for me at all. Four times the price of Fable, and the waiting time is forever.

Cache hit rate on the byte-identical prompt sent minutes earlier: zero. Not low. Zero.

Conclusion

Chinese models reason more - this is the main cause of their slowness.
Per token they are slower too, but only by about 20-40%. Kimi K3 and Qwen 3.8 Max are the exceptions at roughly 2x, and Qwen 3.7 Flash is the fastest generator in the whole roster. The token count does most of the damage, not the throughput.
Some Chinese models are cheap. Kimi K3 and Qwen 3.8 Max are not so cheap.

And this is why I hate benchmarks.

One late addition: while I was writing this, DeepSeek emailed me about a billing adjustment. Continue using the service and you accept the new terms. So the cheapest column in my tables has a shelf life, though I doubt it moves enough to change the ranking.

What Is Latency vs Bandwidth? A Simple Explanation

Sat, 14 Mar 2026 12:55:03 +0000

Why Does Your Internet Feel Slow Even When the Speed Is High?

Have you ever run a speed test and noticed fast internet speeds, yet your video calls still lag or online games feel delayed? It’s frustrating, right?

Many people believe that internet speed alone affects performance, but that’s just part of the picture. Two main factors shape your online experience: latency and bandwidth.

Understanding latency and bandwidth can help you see why your internet sometimes feels slow, even when it shouldn’t. Whether you're streaming movies, working from home, gaming online, or browsing the web, these two concepts directly influence how smoothly everything runs.

In this guide, we’ll explain what latency is, what bandwidth means, how they differ, and why both are important, using simple language.

What Is Bandwidth?

Bandwidth refers to the maximum amount of data that can be transferred over an internet connection within a specific period of time.
In simple terms, bandwidth is the capacity of your internet connection.

It is usually measured in:

Mbps (Megabits per second)
Gbps (Gigabits per second)

Think of Bandwidth Like a Highway

Imagine a highway with multiple lanes.

A wide highway with many lanes allows more cars to travel at once.
A narrow, one-lane highway limits the number of cars that can pass.

Similarly, higher bandwidth allows more data to be transferred simultaneously, helping multiple devices and applications run smoothly.

Examples of Bandwidth in Everyday Internet Use

Higher bandwidth is helpful when:

Streaming 4K videos
Downloading large files
Uploading content to the cloud
Multiple people using Wi-Fi at the same time
Smart homes with many connected devices

For example, if a household has 10 devices connected to Wi-Fi, a higher-bandwidth connection ensures each device receives sufficient data.

What Happens When Bandwidth Is Low?

When bandwidth is limited, you may notice:

Slow downloads
Video buffering
Reduced streaming quality
Slow website loading during peak hours

However, bandwidth alone does not determine responsiveness, which is where latency comes in.

What Is Latency?

Latency is the time it takes for data to move from your device to a server and back. In simple terms, it measures how quickly a network responds after you send a request.

Latency is usually measured in milliseconds (ms). A lower number means a faster response time.

While bandwidth measures how much data can be transferred at once, latency measures the delay before that transfer starts. Both factors affect overall internet performance, but in different ways.

Think of Latency Like Travel Time

A simple way to understand latency is through a highway analogy.
If bandwidth is the width of the highway and determines how many cars can travel at once, latency is the time it takes a car to get from point A to point B.

Even if the highway is wide and lets many cars move simultaneously (high bandwidth), long distances, traffic lights, or congestion can still slow down the trip (high latency). The same idea applies to internet data moving across networks.

Examples of Latency in Everyday Internet Use

Latency is most noticeable during activities that depend on real-time communication or quick responses. In these cases, even tiny delays can impact the user experience.

Common examples include:

Online gaming
Video calls and conferencing
Voice calls over the internet.
Live streaming
Remote work tools and collaboration platforms

When latency is high, users may experience:

Delays during video calls or meetings
Characters responding slowly in online games
Lag in cloud-based applications.
Slow reaction times in remote desktop sessions

These delays happen because the data takes longer to travel between devices and servers.

What Is Considered Good Latency?

Latency can vary depending on the type of connection and the distance to the server. However, general benchmarks include:

0–20 ms: Excellent – ideal for gaming and real-time applications
20–50 ms: Very good performance
50–100 ms: Acceptable for most internet activities
100+ ms: Noticeable delay, which may affect performance

In most cases, lower latency results in faster and smoother online experiences, especially for tasks that require immediate responses.

Latency vs Bandwidth: What’s the Difference?

Although they’re related to internet performance, latency and bandwidth measure completely different things.
Here’s a simple comparison:

Meaning: Bandwidth refers to the amount of data that can be transferred over a network in a certain amount of time. Latency measures how long it takes for data to travel from your device to a server and back.
Measurement: Bandwidth is usually measured in Mbps (Megabits per second) or Gbps (Gigabits per second). Latency, on the other hand, is measured in milliseconds (ms).
Focus: Bandwidth is about network capacity. It tells us how much information can move at once. Latency is about response time. It shows how quickly a connection reacts.
Best For: High bandwidth is important for activities like video streaming, large downloads, and transferring big files. Low latency matters for online gaming, video calls, and real-time applications.
Problem When Low/High: When bandwidth is low, users may face slow downloads, buffering videos, and limited data transfer speeds. When latency is high, users might notice lag, delayed responses, and interruptions in real-time communication.

Quick Summary

Bandwidth = how much data can move
Latency = how fast data responds

Both are important for a smooth internet experience.

Why Latency and Bandwidth Both Matter

Many people look only at internet speed plans that promote higher Mbps numbers. While bandwidth matters, overall internet performance relies on both enough bandwidth and low latency.

Bandwidth decides how much data can be transferred at once, and latency affects how quickly that data starts to move. When both factors work well together, users enjoy smoother and faster internet performance.
Let’s look at a few real-world examples.

1. Online Gaming

Gaming requires very low latency because actions must happen instantly.
Even if your connection has high bandwidth, high latency can still cause:

Lag during gameplay
Delayed actions from controllers or keyboards
A poor and frustrating gaming experience

For competitive or fast-paced games, latency often matters more than bandwidth.

2. Video Streaming

Streaming services rely primarily on bandwidth because large amounts of video data must be delivered continuously.

Higher bandwidth allows:

Smooth HD streaming
Stable 4K video playback
Less buffering interruptions

Latency usually has less impact on streaming because videos are buffered ahead of playback.

3. Video Calls

Video calls require a balance of both bandwidth and latency to work effectively.

They need:

Enough bandwidth to maintain video quality
Low latency to ensure conversations feel natural and real-time

If latency is high, conversations can feel out of sync, causing delays between when someone speaks and when others hear the response.

4. Smart Homes and Multiple Devices

Modern homes often have many connected devices, such as smartphones, smart TVs, security cameras, gaming consoles, and smart home assistants.

In these settings, higher bandwidth matters because multiple devices share the same internet connection. More bandwidth ensures that each device gets enough data without slowing down the entire network.

What Causes High Latency?

Several factors can increase latency and slow down response times.

1. Long Distance to Servers

The farther your device is from the server you're connecting to, the longer it takes for data to travel back and forth.

2. Network Congestion

When many users share the same network, traffic can become crowded, increasing delays in data transmission.

3. Poor Routing

Sometimes, internet data travels through inefficient network paths before reaching its destination, which adds unnecessary delay.

4. Wireless Interference

Wi-Fi interference from nearby devices, walls, or other wireless networks can slow communication between your device and router.

5. Older Network Technology

Outdated network infrastructure may process data less efficiently than modern systems, thereby increasing latency.

What Affects Bandwidth?

Bandwidth availability can also be influenced by several factors.

1. Internet Plan:

Your internet service provider (ISP) determines the maximum bandwidth available through your service plan.

2. Number of Connected Devices:

More devices on the same network means the available bandwidth must be shared among them.

3. Router Quality:

Older or lower-quality routers may struggle to handle high-speed connections or multiple devices simultaneously.

4. Network Congestion:

During peak internet usage hours, network traffic may reduce the effective bandwidth available to users.

How to Improve Latency and Bandwidth

If your internet feels slow or inconsistent, a few simple steps can help improve performance.

Improve Latency

Use wired Ethernet connections instead of Wi-Fi when possible.
Connect to servers that are geographically closer.
Reduce background applications that consume network resources.
Upgrade outdated networking hardware.

Improve Bandwidth

Upgrade to a faster internet plan.
Use modern routers that support higher speeds.
Limit the number of active devices during heavy usage.
Schedule large downloads during off-peak hours.

Improving both latency and bandwidth creates a faster, smoother, and more stable internet experience for everything from gaming to streaming to remote work.

Why Understanding Latency vs Bandwidth Matters

As internet use grows with remote work, streaming, cloud computing, and smart homes, it becomes increasingly important to understand the difference between latency and bandwidth.

When you understand these terms, you can:

Diagnose internet performance issues.
Choose better internet plans.
Optimize your network setup.
Improve streaming, gaming, and work experiences.

Simply put, a fast internet connection is not just about speed, it’s about responsiveness and capacity working together.

Conclusion

Knowing about latency and bandwidth helps explain why internet performance can vary, even with high-speed connections.

Bandwidth determines how much data can travel through your connection, while latency measures how quickly that data responds. Both are key for a smooth online experience.

High bandwidth allows multiple devices and applications to run without slowdowns, while low latency keeps real-time interactions, like gaming and video calls, responsive and seamless.

By understanding how these two factors work together, you can optimize your internet setup, troubleshoot performance issues, and choose the right connection for your needs.

Frequently Asked Questions (FAQs)

1. What is the main difference between latency and bandwidth?

Bandwidth measures the amount of data that can be transferred, while latency measures how long it takes for data to travel between devices and servers.

2. Is low latency better or worse?

Low latency is better because it means faster response times, which is important for gaming, video calls, and real-time applications.

3. Does higher bandwidth reduce latency?

No, higher bandwidth does not automatically reduce latency. They are separate factors, although both affect overall internet performance.

4. What is good latency for gaming?

Latency under 20–50 milliseconds is considered good for most online games. Higher latency may cause lag.

5. Which is more important: latency or bandwidth?

It depends on the activity. Bandwidth is more important for streaming and downloads, while latency is critical for gaming, video calls, and real-time communication.