Coder Legion - Recent posts tagged speech-to-text

The 500ms Budget That Shapes Every Voice Agent You Build

Tue, 08 Sep 2026 02:00:32 +0000

Most of us here have built a chatbot at some point, and going from text to voice looks like it should just be two more API calls, one on each end. It is not, and the reason is a single latency number that reorganizes the whole pipeline.

A voice agent runs the same loop every turn: capture audio, transcribe it, run it through a model, generate a reply, synthesize speech, play it back. That entire round trip has to land under about 500 milliseconds or the caller hears a gap and assumes the line dropped. A text chatbot can sit there for two or three seconds and nobody blinks, so most of the habits you carry over from text work are suddenly too slow.

That budget is why production voice stacks look the way they do. Speech recognition runs in streaming mode so transcription starts before the caller stops talking, which lets the language model begin working on a partial sentence instead of waiting for a complete one. Modern ASR from Deepgram, AssemblyAI and Google sits under a 5 percent word error rate on clear English, so accuracy is largely a solved problem. The engineering work is all about where the milliseconds go.

The other thing that eats the budget is end of turn detection. Voice activity detection has to decide when you have actually stopped speaking, and that call happens before the model sees a single token. Cut it short and you clip people mid sentence, wait too long and your real time agent feels like a phone tree.

If you want the full pipeline breakdown, the open source options like LiveKit, Pipecat and Vocode, and where the cost actually lands, AI voice agents covers it end to end.

The cost comparison surprised me more than the latency did. A US call center seat runs $25 to $65 an hour once you count benefits, training and management overhead, while a voice agent handling the same calls costs $0.05 to $0.25 a minute with no ramp up and unlimited concurrency. That gap is why enterprise production deployments grew 340 percent year over year through early 2026.

Curious what people here have hit in practice. If you have shipped a voice agent, where did your latency actually go, the ASR, the model, or the TTS?

The New Best AI Subtitle Model? Canary Qwen 2.5B Tested Against Whisper Large V3

Sat, 02 May 2026 02:26:28 +0000

Best Open Source Subtitle Generator? Canary Qwen 2.5B + Whisper Full Guide

Full tutorial link > https://www.youtube.com/watch?v=4lAk6sf1qF8

NVIDIA NeMo Canary-Qwen-2.5B is an English speech recognition model that achieves state-of-the art performance on multiple English speech benchmarks. Canary model is the new king that dethroned famous Whisper.

Full tutorial for the Whisper TTS Premium speech-to-text app by SECourses with new NVIDIA Canary Qwen 2.5B support. In this video, I demo local subtitle generation, compare Canary Qwen 2.5B against Whisper Large V3, show output formats, batch processing, presets, YouTube URL and live microphone options, then install the app from scratch on Windows.

You will also see RunPod and Massed Compute notes, first-run model download, RTX 5000/CUDA 13 driver requirements, subprocess mode for preventing VRAM/RAM leaks, and when to use Whisper instead of Canary.

Links:

Download App and the source post: [ [https://www.patreon.com/posts/whisper-webui-to-145395299](https://www.patreon.com/posts/whisper-webui-to-145395299) ]

Discord: [ [https://discord.com/channels/772774097734074388/1079506787734134844](https://discord.com/channels/772774097734074388/1079506787734134844) ]

Patreon app index: [ [https://github.com/FurkanGozukara/Stable-Diffusion/blob/main/Patreon-Posts-Index.md](https://github.com/FurkanGozukara/Stable-Diffusion/blob/main/Patreon-Posts-Index.md) ]

Related RunPod/Massed Compute setup tutorial: [ [https://youtu.be/ZRrzvD4wNys](https://youtu.be/ZRrzvD4wNys) ]

In my tutorial-video tests, Canary Qwen 2.5B achieved 5.91% global WER and reached up to 46x faster than real-time transcription, making it my new recommended default for English speech-to-text. Whisper remains useful when you need broader spoken-language support or word-level timestamps.

Chapters:

0:00 Intro to the local open-source speech-to-text app and new Canary support
0:20 Quick demo setup with NVIDIA Canary Qwen 2.5B
0:33 Maximum-quality defaults and starting subtitle generation
0:48 Live transcription speed and accuracy preview
1:03 Chunk length settings for smaller or larger subtitle segments
1:18 Fast generation, supported exports, and restarting with all formats
1:31 Multiple subtitle file formats explained
1:47 Batch processing folders, output paths, subfolders, and overwrite mode
1:58 YouTube URLs, microphone/live transcription, translation, and BGM separation
2:09 Saving presets and using advanced parameters
2:24 Auto-optimized defaults for Whisper and Canary models
2:39 Canary Qwen 2.5B vs Whisper Large V3 comparison begins
2:54 Real-world WER benchmark and 5.91% Canary result
3:10 Why non-native English speech is harder to transcribe accurately
3:24 Canary speed advantage and 46x real-time transcription explained
3:43 Test averages across long and short tutorial videos
3:59 Cases where Whisper slightly wins and final Canary recommendation
4:14 Opening the output folder after transcription completes
4:27 VTT output, matching filenames, capitalization, and punctuation
4:44 Accuracy examples inside the generated transcript
4:58 TXT, TSV, SRT, LRC exports and word-level timestamp note
5:20 Download page, latest ZIP, and installation overview
5:31 Windows requirements: Python 3.11, Git, CUDA, and C++ notes
5:51 Choosing install location and keeping the app isolated in venv
6:04 Extracting the ZIP and running Windows install/update BAT
6:23 Automatic model downloads on first run
6:34 RunPod, Massed Compute, and Linux installation files
6:50 Where to learn RunPod and Massed Compute setup in the related guide
7:29 UV-powered Windows installation completes quickly
7:41 Starting the app with Windows start app BAT
7:58 Selecting video/audio input and generating subtitles on a fresh install
8:10 First-run Canary model download and 5GB model size
8:35 Easy setup goal and automatic fresh-install workflow
8:53 Discord, Patreon index, and 100+ SECourses applications
9:13 RTX 5000 support and updated NVIDIA driver requirement
9:35 Fresh-install transcription starts successfully
9:47 Automatic downloads for Canary, Whisper, diarization, and extra tools
10:16 Canary becomes the new default model recommendation
10:36 Subprocess mode to prevent VRAM and RAM leaks
10:51 Why running transcription as a subprocess is recommended
11:04 Switching back to Whisper models when needed
11:20 Whisper language coverage vs Canary and audio/video support
11:42 Real recording benchmark: 27 minutes transcribed in about 2 minutes
11:56 Model loading overhead and clean RAM/VRAM release
12:08 Final notes, subscribe reminder, and downloading the full transcript ZIP

Download Installers and App

https://www.patreon.com/posts/145395299

30 April 2026 - Version 10.0

This is a quite big upgrade to our application
We now fully support NVIDIA NeMo Canary-Qwen-2.5B is an English speech recognition model : https://huggingface.co/nvidia/canary-qwen-2.5b
This model is currently State Of The Art (SOTA) Speech to Text model for English language
I have done extensive research and testing and it is set to best default parameters
Fully supporting all of the features our Whisper app were already supporting
Get the zip file, overwrite all previous files and run installer for update / upgrade
The model will be auto downloaded when you first time run

I also have compared with Whisper best configurations are here the comparison results - best results of Whisper taken

As you can see NVIDIA NeMo Canary-Qwen-2.5B is not only significantly better but also faster

15 April 2026 - Version 8.0

Diarization had some error and this is fixed
Mic tab completey remade and now both live transcription from microphone and offline transcription from microphone working
- Live transcription quality is not that great
- Both live transcription and offline transcription recordings from microphone will be saved in outputs folder
- Live transcription will auto run but for offline transcription first record voice with microphone and then click Generate Subtitles button
Don't forget to select your working microphone and give permission for app to use your microphone from your browser
For update / install get the latest zip file, overwrite older files and run Windows_Install_Update.bat

14 April 2026 - Version 7.0

Now auto downloads Diarization files and thus you don't need to enter Hugging Face token and get permission
Now you can copy paste any YouTube link and generate subtitles
- This was broken and now fixed
- It will save generated files with same name as the video title
Now you can batch generate subtitles for YouTube video channels
Paste the video channel, enable batch and it will generate subtitles for every video
- Set how many videos you want (scans latest ones)
- You may get rate limited by YouTube
For update / install get the latest zip file, overwrite older files and run Windows_Install_Update.bat

8 April 2026 - Version 5.0

This is a massive update with so many new features
- Get the latest zip file and make a fresh install please > https://www.patreon.com/posts/145395299
- 1-Click to install on Windows, RunPod, SimplePod, Massed Compute, Linux
New preset save and load system with extremely well tested best_quality and fast pre-made presets
- Presets are automatically loaded as you change them and also last used preset is remembered when you restart the app
- Word Timestamps is enabled by default to improve quality but it also generates regular version as well automatically
Download transcription button
Open outputs folder button (all transcriptions automatically saved)
Load video / audio file directly from path (useful for platforms like RunPod where Gradio upload is slow)

The fast preset uses new custom in house implemented batch size 32 feature and it is literally blazing fast compared to all other existing Whisper apps and repos
Fully supporting all kind of video and audio formats upload with full preview
Batch folder processing process given folder all files automatically
Live transcription Window that shows latest transcription live while processing
At batch size 1 with best quality, 11x real time transcription speed (depends on GPU)
At batch size 32 fast preset 15x to 30x real time transcription speed (depends on GPU)
New feature Repeat Initial Prompt Every Window

Supports all Whisper models like Large V1, Large V3, Turbo, Distill Large, Tiny, etc
Supports following format outputs you can have checked all so all generated at the same time : SRT, WebVTT, txt, LRC,JSON, TSV
- All outputs will have the same name as your input file name
With sub process working system, you can cancel any processing immediately with 0 RAM or VRAM leak
Fully supports Windows and Linux (use Massed Compute installer)
Based on Python 3.11 VENV and CUDA 13 and Torch 2.9.1 with pre-compiled libraries like Flash Attention
If you don't like output, try to enable / disable Condition On Previous Text it makes big difference

The app supports 100 languages and 32 models

Lots of Advanced Parameters and all set to best quality
Built in Background Music Remover Filter
Built in Voice Detection Filter
Fully detailed CMD output to watch entire progress
Extremely optimized VRAM usage as low as 6 GB GPUs

Some other utility features like YouTube, record from a Mic, T2T Translation, BGM Seperation

Full Page Screenshot

The Transformative Impact of AI in Gamified Formats: Revolutionizing Healthcare Systems and Language

Sun, 07 Sep 2025 07:16:03 +0000

Introduction

In an era where artificial intelligence (AI) is reshaping every facet of human life, its integration into gamified formats stands out as a particularly innovative approach. Gamification, the application of game-design elements in non-game contexts, leverages AI to make complex tasks engaging, educational, and effective. This is especially evident in healthcare systems, where AI-powered games are aiding in disease research, patient engagement, and treatment adherence. Similarly, in language translation, AI-driven tools are breaking down barriers, particularly for underrepresented African languages, enabling better communication, cultural preservation, and access to global knowledge.
This post explores the profound impact of AI in these domains, focusing on gamified applications that aid healthcare—such as citizen science games for cancer research—and AI models for translating African languages alongside other global tongues. We will cite key researchers, including Dr. Odetunji Ajadi Odejobi from Obafemi Awolowo University (OAU) in Nigeria, whose work on Yoruba language processing exemplifies the potential of AI in low-resource languages. Other professors and experts, such as those involved in AI initiatives at OAU and global projects, will also be highlighted. Drawing from recent studies and developments, this comprehensive analysis underscores how AI gamification is not just a trend but a game-changer for equitable health and linguistic inclusivity.
The discussion is timely, as AI's role in healthcare gamification has been shown to boost patient engagement by up to 30% in medication adherence. 7 Meanwhile, for language translation, initiatives like Lelapa AI's InkubaLM are pioneering multilingual models for African languages, addressing the digital divide. 24 Over the next sections, we'll delve into these impacts, supported by evidence from over 50 sources, aiming for a holistic view that spans approximately 3000 words.

Section 1: AI Gamification in Healthcare – A New Paradigm for Engagement and Outcomes
Gamification in healthcare involves incorporating elements like points, badges, leaderboards, and challenges into medical applications, often powered by AI for personalization and adaptability. This approach transforms mundane health tasks into interactive experiences, improving outcomes in prevention, treatment, and research. AI enhances this by analyzing user data in real-time, adjusting difficulty levels, and providing tailored feedback, making interventions more effective.
One key impact is on patient engagement. Traditional healthcare often struggles with adherence; for instance, non-compliance with medication regimens affects up to 50% of patients worldwide. Gamified apps, however, use AI to create rewarding loops. A study on gamified health apps found a 30% improvement in adherence compared to non-gamified ones. 7 Platforms like Smartico.ai emphasize how interactive features make learning about health enjoyable, leading to better retention of information.
In mental health, the convergence of AI and gamification is revolutionary. AI algorithms adapt game scenarios to users' emotional states, offering personalized therapy. For example, gamified cognitive behavioral therapy apps use AI to track mood patterns and suggest interventions, reducing symptoms of anxiety and depression. 6 A scoping review on gamification for clinical reasoning education highlights its role in enhancing decision-making skills among healthcare providers, with AI enabling adaptive simulations. 5
Business benefits are also significant. Healthcare providers using gamification see improved patient retention and cost savings. OpenLoyalty.io notes that strategic implementation boosts treatment adherence, reducing readmissions. 12 In Africa, where resource constraints are acute, AI-gamified tools could address workforce shortages. A 2025 study on Nigerian oncologists' views on AI shows optimism for its role in diagnostics, which could extend to gamified training modules. 3
Professor Celestine Iwendi, affiliated with OAU through collaborations, heads the Centre of Intelligence of Things (CIoTh) and emphasizes AI's potential in sustainable health solutions. 36 His work on AI and IoT could integrate with gamification for remote monitoring in African contexts.
Challenges include ensuring accessibility and avoiding over-reliance on technology. Ethical concerns, like data privacy in AI-driven games, must be addressed. Nonetheless, the rise of health gamification is undeniable, with impacts on engagement that could save lives.
Section 2: AI Games Specifically Aiding Cancer Research and Healthcare Systems
Focusing on cancer, AI games are crowdsourcing solutions to complex problems. These "serious games" turn research into playable challenges, accelerating discoveries.

A prime example is GENIGMA, launched in 2022 by Spanish researchers. Players solve puzzles to map genomic sequences in cancer cell lines like T47D for breast cancer, creating high-resolution genome maps. 16 Over 500,000 solutions from players have advanced understanding of cancer mutations. 16
Similarly, AcCELLerate, developed with Dr. Priyanka Bhosale at King's College London, has players trace dye-stained tongue cells to train AI for oral cancer detection. 0 This gamified approach speeds up tumor assessment, with potential for other cancers.
The National Cancer Institute's PERCEPTION uses AI for drug response prediction, but games like Foldit and Eterna generate data for oncology. 15 Foldit, where players fold proteins, has led to breakthroughs in enzyme design for cancer therapies. 15
AI supercomputing, like Argonne's Aurora, screens billions of molecules for cancer inhibitors in minutes. 18 Integrating this with games could democratize research.
In Nigeria, while no specific AI game by a local doctor is documented, OAU's AI ecosystem, led by Professor G. A. Aderounmu in the EmbeddedAI program, supports health applications. 36 Dr. Odejobi's pattern recognition expertise—e.g., Yoruba handwriting recognition with 87.7% accuracy using KNN—could adapt to cell image analysis in cancer games. 36 39
Hurone AI by Dr. Kingsley Ndoh focuses on cancer control in Africa, potentially gamifiable. 0 Gaming inspires AI strategies, like battle royale ideas for immunotherapy. 22

These games foster public understanding and healthy lifestyles, impacting healthcare systems by reducing burdens through early detection.

Section 3: AI's Role in Language Translation for African and Other Languages
African languages, numbering over 2000, are often low-resource, lacking data for AI training. AI translation bridges this gap, enabling access to education, healthcare, and commerce.
Google Translate supports 25 African languages, including Yoruba and Igbo. 28 But for deeper accuracy, specialized models are needed. Lelapa AI's InkubaLM is Africa's first multilingual LLM for low-resource languages
The African Languages Lab delivers context-aware translations using multimodal datasets. 25 Scientists are recording 9000 hours of Kenyan, Nigerian, and South African languages for AI training.
Botlhale AI connects African populations to businesses via translation. 29 Sunbird AI translates five Ugandan languages. 33 OBTranslate aims for 2000+ African languages.
For other languages, AI like DeepL and neural models handle high-resource ones, but African LLMs offer opportunities in education and healthcare.
Dr. Odejobi's research on Yoruba tone recognition using ANN and fuzzy logic is pivotal. 42 45 His work on diacritic restoration and sequence-to-sequence learning enhances translation accuracy for tonal languages. 44 As a senior lecturer at OAU, his contributions to speech engineering could integrate with gamified language apps.
Professor Iyabo Olukemi Awoyelu at OAU specializes in data mining, aiding dataset creation for translation.
In healthcare, AI translation ensures inclusive care. Universal Voice Translation models support African languages in emergencies.
X posts highlight community efforts, like SaharaLabsAI supporting 76 African languages. 46 Vambo AI offers translation in 44 languages. 56
Challenges include data scarcity and hallucinations in LLMs, but projects like EqualyzAI preserve cultural essence. 53

Section 4: Intersections – Gamified AI for Healthcare and Language Translation

The overlap is promising: Gamified AI could teach languages in healthcare contexts, like apps for medical terminology in African languages. OAU's AI training programs could develop such tools. 36
In cancer care, translated gamified apps could engage diverse populations. Dr. Odejobi's language work could adapt games like GENIGMA for Yoruba speakers.

Global trends show AI games inspiring cross-domain innovations. 19
AI in gamified formats is profoundly impacting healthcare and language translation. From cancer games accelerating research to models preserving African languages, the potential is vast. Citing experts like Dr. Odejobi, whose Yoruba AI research paves the way for inclusive tools, and professors at OAU driving embedded AI, we see a future of equity.
As initiatives expand, collaboration is key. The African Next Voices project and others are closing gaps. 23 Ultimately, AI gamification isn't just play—it's progress

Breaking the Language Barrier in Healthcare—One Whisper at a Time

Mon, 14 Jul 2025 17:27:26 +0000

Ever sat in a Bangladeshi clinic and heard someone say something like:

“Apni aagei bolen chilen na je chest-e ekta tightness feel hocche?”

That right there is code-switching—slipping between Bengali and English mid-sentence, like it’s no big deal. It happens all the time, especially in doctor-patient conversations. But while humans handle this effortlessly, most language models? They get confused faster than a junior doctor on their first ER shift.

And in healthcare, that’s a problem. You can’t afford to misinterpret symptoms because a model panicked at the word "tightness" sandwiched between two Bengali clauses.

So I decided to do something about it.

Enter: MediBeng Whisper Tiny

I built a small but mighty tool: a fine-tuned version of OpenAI’s Whisper Tiny model trained specifically to handle Bengali-English code-switched medical speech and translate it into clean English.

Yep, that means this model can listen to real mixed-language clinical convos and output clear, usable English—perfect for documentation, analysis, or even aiding with diagnosis.

Why I Did This

In real-world clinics across Bangladesh (and frankly, anywhere multilinguals exist), people constantly mix languages. It’s second nature. But no existing ASR model was cutting it when it came to understanding this hybrid speech, let alone translating it. The goal was simple:

Input: Bengali-English speech
Output: Clear English transcription

And for this? I had to train Whisper to stop freaking out every time it heard a Bengali sentence ending with an English medical term.

How I Did It (a.k.a. The Nerdy Bit)

Built a Custom Dataset – MediBeng

I couldn’t find a dataset that captured the quirky, authentic blend of Bengali and English that you hear in clinics. So I built my own: MediBeng.

It’s a synthetic but realistic dataset of code-switched speech based on how real people actually talk in clinical settings.

Fine-Tuned Whisper Tiny

I used MediBeng to fine-tune Whisper Tiny, teaching it how to make sense of bilingual speech. That means adjusting the model's brain (aka weights) to understand when someone says:

“Take korar por ekdom bhalo feel kori nai.”

...they probably meant something worth documenting.

✅ 3. Tested It on Real Conversations

After fine-tuning, I ran the model on real examples—and the improvement was obvious. It could now transcribe and translate code-switched speech much more accurately, giving output that was actually usable for patient records.

Why This Actually Matters

This isn’t just a fun project or an academic exercise—it solves a real problem. Healthcare professionals are already overwhelmed. Asking them to decode bilingual gibberish in transcripts just makes their job harder.

With MediBeng Whisper Tiny, we’re talking:

Fewer transcription errors

Better documentation

Faster decision-making

Happier doctors (and probably patients too)

The Big Picture

This project proves that even lightweight models like Whisper Tiny can be smart—if you give them the right data. It also shows the massive potential of domain-specific, culturally aware AI tools.

Imagine tailoring models like this for other regions: Hindi-English, Arabic-French, Spanish-Spanglish—you get the idea.

I started with Bengali-English medical speech because that’s where the pain was. But this blueprint? It’s open-source, replicable, and ready to be extended by anyone who wants to build smarter speech models for multilingual communities.

Check out the repo to dive into the code or replicate this yourself:

MediBeng Whisper Tiny Model

MediBeng Dataset

GitHub Repository

TL;DR

People code-switch. A lot. Especially in clinics.

ASR models weren’t handling it well.

So I trained Whisper Tiny on Bengali-English medical speech.

It now translates mixed speech into clean English.

It’s free and open for you to build on.

If you’re working in multilingual healthcare tech—or just love solving messy real-world AI problems—this one’s for you.

Let’s make speech models speak our language(s).