Coder Legion - Recent posts tagged audio

Building a Low-Latency Virtual Piano and Drum Kit on iOS with AVAudioEngine

Tue, 27 Oct 2026 12:40:37 +0000

Originally published at https://bestlatestapps.com/blog/building-a-low-latency-virtual-piano-and-drum-kit-on-ios-with-avaudioe/

Samplers for pitched instruments, a player-node pool for drums, a multi-touch keyboard that handles chords and glissandos, and note names for beginners

The test for a virtual instrument is brutal and immediate: tap a key, and the sound has to arrive with your finger, not after it. Play a chord and you expect three notes, not one. Drag your finger along the keyboard and you expect a glissando, not silence. Hit the snare four times quickly and you expect four hits, not one hit cut off three times.

My app Drums n Piano puts a piano, a drum kit and a xylophone on your phone. The listing promises an instrument that "sounds real, just like the original instrument!", and it's also meant for learning: it teaches beginners "the names & positions of notes on the piano".

This article covers how I'd build the audio and touch parts of an app like that on iOS. That means configuring the session for low latency, using AVAudioUnitSampler for pitched instruments, a pool of AVAudioPlayerNodes for one-shot drum samples, a custom multi-touch keyboard view, and labelling notes for learners. The app supports iOS 12 and later, so the touch layer is UIKit, and I'll explain why that's the right call even today. API names and availability are checked against Apple's documentation, and this is the approach I'd use rather than a tour of the app's source.

1. The three things a virtual instrument must get right

Latency. The gap between touch and sound. Part of it is the audio buffer, and part is your own code: loading a file on tap, or allocating while the user is playing.
Polyphony. Several sounds at once: chords on the piano, overlapping cymbal and kick, a fast roll on the snare.
Touch handling. Many fingers at once, sliding between keys, and no gesture delays.

None of these are hard individually. They get hard when the first prototype plays a sound file with AVAudioPlayer on each tap. That loads, decodes and allocates on the main thread, can't overlap itself, and has noticeably more delay than the engine-based approaches below.

2. Session setup for an instrument

import AVFoundation

enum InstrumentSession {
    static func activate() throws {
        let session = AVAudioSession.sharedInstance()
        // .playback: the instrument IS the app, so play even with the Silent switch on.
        // .mixWithOthers: let people play along with music from another app.
        try session.setCategory(.playback, mode: .default, options: [.mixWithOthers])
        try session.setPreferredIOBufferDuration(0.005)   // ask for ~5 ms; the system decides
        try session.setActive(true)
        print("IO buffer: \(session.ioBufferDuration * 1000) ms, rate: \(session.sampleRate) Hz")
    }
}

Two decisions here deserve a sentence each:

.playback vs .ambient. .ambient respects the Silent switch, which suits a game's background music. For an instrument, silence when the switch is on looks like a broken app, so I use .playback.
A small preferred buffer reduces output latency. It's a preference: read back ioBufferDuration to see what you actually got, and expect Bluetooth headphones to add delay you can't remove. Wired headphones or the built-in speaker are best for playing in time.

3. Pitched instruments: AVAudioUnitSampler

For piano and xylophone, you don't want one audio file per tap. You want a sampler: an instrument that receives MIDI-style "note on/off" messages and plays the right sample, pitch-shifting between recorded notes if needed. AVAudioUnitSampler (iOS 8+) is Apple's sampler, wrapped as an engine node.

import AVFoundation

final class PitchedInstrument {
    private let engine: AVAudioEngine
    private let sampler = AVAudioUnitSampler()

    init(engine: AVAudioEngine) {
        self.engine = engine
        engine.attach(sampler)
        engine.connect(sampler, to: engine.mainMixerNode, format: nil)
    }

    /// Load a SoundFont (.sf2) or DLS bank shipped in the bundle.
    func loadSoundBank(named name: String, program: UInt8 = 0) throws {
        guard let url = Bundle.main.url(forResource: name, withExtension: "sf2") else {
            throw CocoaError(.fileNoSuchFile)
        }
        try sampler.loadSoundBankInstrument(
            at: url, program: program,
            bankMSB: UInt8(kAUSampler_DefaultMelodicBankMSB),
            bankLSB: UInt8(kAUSampler_DefaultBankLSB))
    }

    /// Or: load individual samples (e.g. one recording every few semitones).
    func loadSamples(_ urls: [URL]) throws {
        try sampler.loadAudioFiles(at: urls)
    }

    func noteOn(_ midi: UInt8, velocity: UInt8 = 100) {
        sampler.startNote(midi, withVelocity: velocity, onChannel: 0)
    }

    func noteOff(_ midi: UInt8) {
        sampler.stopNote(midi, onChannel: 0)
    }
}

Why this is the right tool:

Polyphony is built in. Ten fingers, ten notes. The sampler manages voices for you.
Note-off is real. Releasing a key triggers the sample's release, as on a real piano, rather than a hard cut.
Loading happens once, at launch or when the user switches instrument, never on the tap.

Where do the sounds come from? Either a SoundFont you have the rights to ship, or your own recordings, with one sample every few semitones loaded via loadAudioFiles(at:). Sample quality is the single biggest factor in "sounds real", and it's a licensing question as much as a technical one: check the licence of any sound bank before you bundle it.

4. Drums: preloaded buffers and a player-node pool

Drums are different. There's no pitch, every hit is a one-shot, and the same pad is hit repeatedly and fast. You could map drums onto a sampler (General MIDI percussion is exactly that), but a simple, very controllable approach is:

Decode each drum sample into an AVAudioPCMBuffer once, at launch.
Keep a small pool of AVAudioPlayerNodes per pad.
On each hit, schedule the buffer on the next node in the pool, round-robin.

import AVFoundation

final class DrumKit {
    private let engine: AVAudioEngine
    private var buffers: [String: AVAudioPCMBuffer] = [:]
    private var pools: [String: [AVAudioPlayerNode]] = [:]
    private var nextIndex: [String: Int] = [:]
    private let voicesPerPad = 4     // how many overlapping hits per pad

    init(engine: AVAudioEngine) { self.engine = engine }

    /// Call once, before starting the engine.
    func load(pads: [String]) throws {
        for pad in pads {
            guard let url = Bundle.main.url(forResource: pad, withExtension: "caf") else {
                throw CocoaError(.fileNoSuchFile)
            }
            let file = try AVAudioFile(forReading: url)
            guard let buffer = AVAudioPCMBuffer(pcmFormat: file.processingFormat,
                                                frameCapacity: AVAudioFrameCount(file.length)) else {
                throw CocoaError(.fileReadCorruptFile)
            }
            try file.read(into: buffer)
            buffers[pad] = buffer

            pools[pad] = (0..<voicesPerPad).map { _ in
                let node = AVAudioPlayerNode()
                engine.attach(node)
                engine.connect(node, to: engine.mainMixerNode, format: buffer.format)
                return node
            }
            nextIndex[pad] = 0
        }
    }

    /// Call after engine.start().
    func startNodes() {
        pools.values.flatMap { $0 }.forEach { $0.play() }
    }

    func hit(_ pad: String, velocity: Float = 1.0) {
        guard let buffer = buffers[pad], let pool = pools[pad], let i = nextIndex[pad] else { return }
        let node = pool[i]
        nextIndex[pad] = (i + 1) % pool.count
        node.volume = velocity
        // .interrupts: if this voice is still ringing from 4 hits ago, restart it cleanly.
        node.scheduleBuffer(buffer, at: nil, options: .interrupts, completionHandler: nil)
    }
}

The whole setup:

let engine = AVAudioEngine()
let piano = PitchedInstrument(engine: engine)
let drums = DrumKit(engine: engine)

try InstrumentSession.activate()
try piano.loadSoundBank(named: "Piano")
try drums.load(pads: ["kick", "snare", "hihat-closed", "hihat-open", "crash", "tom-high", "tom-low"])
engine.prepare()
try engine.start()
drums.startNodes()

Why a pool? A single player node plays one schedule queue. Scheduling a new hit with .interrupts on a single node cuts the previous hit off, and that's audible on cymbals. With four voices per pad, four overlapping hits all ring naturally, and only the fifth recycles the oldest voice. Ship samples as uncompressed .caf or .wav so decoding at launch is quick and exact. They're short, so the size cost is small.

5. A multi-touch keyboard that feels like a keyboard

This is where many SwiftUI prototypes fall short. A Button per key fires on touch-up, which is far too late for an instrument. A DragGesture per key doesn't hand off to the next key when the finger slides. And coordinating several simultaneous fingers through SwiftUI gestures is awkward.

The direct approach is one UIKit view for the whole keyboard that tracks every UITouch itself:

import UIKit

final class KeyboardView: UIView {
    var firstMidi: UInt8 = 60           // C4
    var octaves = 2
    var onNoteOn: ((UInt8) -> Void)?
    var onNoteOff: ((UInt8) -> Void)?

    private var touchNotes: [ObjectIdentifier: UInt8] = [:]   // which finger holds which note

    override init(frame: CGRect) {
        super.init(frame: frame)
        isMultipleTouchEnabled = true      // essential: off by default
    }
    required init?(coder: NSCoder) { fatalError("init(coder:) is not supported") }

    // MARK: Touch tracking

    override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
        for t in touches { press(t) }
    }

    override func touchesMoved(_ touches: Set<UITouch>, with event: UIEvent?) {
        for t in touches {
            let id = ObjectIdentifier(t)
            let newNote = note(at: t.location(in: self))
            guard newNote != touchNotes[id] else { continue }    // still on the same key
            if let old = touchNotes[id] { onNoteOff?(old) }       // glissando: release old…
            touchNotes[id] = newNote
            if let n = newNote { onNoteOn?(n) }                   // …and press new
            setNeedsDisplay()
        }
    }

    override func touchesEnded(_ touches: Set<UITouch>, with event: UIEvent?) {
        for t in touches { release(t) }
    }

    override func touchesCancelled(_ touches: Set<UITouch>, with event: UIEvent?) {
        for t in touches { release(t) }     // never leave a note stuck on
    }

    private func press(_ t: UITouch) {
        guard let n = note(at: t.location(in: self)) else { return }
        touchNotes[ObjectIdentifier(t)] = n
        onNoteOn?(n)
        setNeedsDisplay()
    }

    private func release(_ t: UITouch) {
        if let n = touchNotes.removeValue(forKey: ObjectIdentifier(t)) { onNoteOff?(n) }
        setNeedsDisplay()
    }

    // MARK: Geometry

    private static let whiteOffsets: [UInt8] = [0, 2, 4, 5, 7, 9, 11]          // C D E F G A B
    private static let blackOffsets: [(after: Int, semitone: UInt8)] =
        [(0, 1), (1, 3), (3, 6), (4, 8), (5, 10)]                               // C# D# F# G# A#

    private var whiteKeyWidth: CGFloat { bounds.width / CGFloat(7 * octaves) }

    /// Black keys sit on top, so test them first.
    func note(at p: CGPoint) -> UInt8? {
        guard bounds.contains(p) else { return nil }
        let w = whiteKeyWidth
        if p.y < bounds.height * 0.6 {
            for octave in 0..<octaves {
                for (after, semitone) in Self.blackOffsets {
                    let x = CGFloat(octave * 7 + after + 1) * w - w * 0.3
                    if CGRect(x: x, y: 0, width: w * 0.6, height: bounds.height * 0.6).contains(p) {
                        return firstMidi + UInt8(octave * 12) + semitone
                    }
                }
            }
        }
        let index = Int(p.x / w)
        let octave = index / 7, step = index % 7
        return firstMidi + UInt8(octave * 12) + Self.whiteOffsets[step]
    }
}

The essential details:

isMultipleTouchEnabled = true. It's false by default, and without it you'll only ever get one finger.
Track notes per touch with ObjectIdentifier(touch). Each finger owns its note, so chords and glissandos work independently.
Handle touchesCancelled. A system gesture or an incoming call can cancel touches. If you don't release notes there, you get stuck notes.
Black keys are hit-tested first, because they're drawn on top.

On iOS 12 this is the only option, since SwiftUI needs iOS 13. Even on a SwiftUI app today, I'd wrap this view in a UIViewRepresentable. Touch handling for an instrument is one of the places where UIKit is still simply the better tool.

Drum pads follow the same idea in simpler form: a grid of rectangles, touchesBegan triggers hit(_:), and there's no note-off.

6. Teaching note names and positions

The listing describes the app as "teaching beginners the names & positions of notes on the piano". On the keyboard view, that's two small things: draw a label on each key, and let learners switch between naming systems.

enum NoteNaming: String, CaseIterable {
    case letters      // C D E F G A B
    case solfege      // Do Re Mi Fa Sol La Si

    static let letterNames = ["C", "C♯", "D", "D♯", "E", "F", "F♯", "G", "G♯", "A", "A♯", "B"]
    static let solfegeNames = ["Do", "Do♯", "Re", "Re♯", "Mi", "Fa", "Fa♯", "Sol", "Sol♯", "La", "La♯", "Si"]

    func name(for midi: UInt8, withOctave: Bool = true) -> String {
        let pitchClass = Int(midi) % 12
        let base = self == .letters ? Self.letterNames[pitchClass] : Self.solfegeNames[pitchClass]
        return withOctave ? "\(base)\(Int(midi) / 12 - 1)" : base      // MIDI 60 = C4
    }
}

// NoteNaming.letters.name(for: 60) == "C4"; NoteNaming.solfege.name(for: 64) == "Mi4"

In draw(_:), draw white-key labels near the bottom of each key and black-key labels only when a "show sharps" toggle is on; for beginners, too many labels is noise. Highlighting the pressed key matters more than any label. The immediate visual "this is the key you just played" is how position is learned.

7. Real-world example: Drums n Piano

This is the approach I'd use, and recommend, for Drums n Piano.

At the time of writing, its App Store listing shows:

Free, in Music and Entertainment
First released in September 2022
9 ratings in the US store, averaging about 4.4 stars
Supports iOS 12.0 and later

How the listing maps to this article:

What the listing says	Where it's covered
"Simply play and learn the piano, drums or xylophone wherever you are"	A sampler per pitched instrument; a player-node pool for drums (sections 3–4)
"This virtual app sounds real, just like the original instrument!" / "Premium sound quality"	Real samples loaded once, a low-latency session, natural note-off releases (sections 2–3)
"Teaching beginners the names & positions of notes on the piano"	Note labels and pressed-key highlighting (section 6)
"Play or learn your music chords"	Per-touch tracking with multi-touch enabled (section 5)

The listing's audience is broad: "a music teacher, singer, songwriter, or beginner or joyful kid". That's a good reminder that kids mash several keys with their whole hand. A multi-touch view that never leaves a stuck note is worth more than any feature.

8. Before you ship a virtual instrument: a checklist

Nothing loads on tap. Samples and sound banks load at launch or on instrument switch; taps only trigger notes.
The session is .playback (or a deliberate .ambient), with a small preferred buffer, and you've logged what you actually got.
Polyphony is tested. Ten-finger chords on the piano; rapid repeated hits on every drum pad.
There are no stuck notes. Test cancelled touches, a phone call mid-chord, backgrounding mid-note and Control Center swipes.
isMultipleTouchEnabled is on, and each touch owns its note.
Glissando works: sliding across keys releases and presses notes as you go.
Sample licences are cleared for everything you bundle.
You've tested on real hardware with wired and Bluetooth audio, and the app tells users that wired is tighter.

Wrapping up

A virtual instrument is mostly preparation: decode everything up front, pick the right player for the job (a sampler for pitched instruments, a node pool for one-shots), keep the audio buffer small, and track every finger yourself. Get those right, and the sound arrives with the touch, which is all a player really asks for.

Play it, or explore my other apps:

Drums n Piano (iOS): https://apps.apple.com/us/app/drums-n-piano/id1642061667
All of my free iOS apps: https://apps.apple.com/us/developer/alok-singh/id1508795536
My portfolio: https://bestlatestapps.com/alok/

If you've built an instrument app and found a better way to handle touches or voices, tell me about it in the comments. I'm especially curious about anyone doing velocity from touch force or radius.

— Alok Kumar Singh

Originally published at https://bestlatestapps.com/blog/building-a-low-latency-virtual-piano-and-drum-kit-on-ios-with-avaudioe/

Building a Real-Time Mic-to-Speaker App on iOS with AVAudioEngine

Tue, 27 Oct 2026 12:40:22 +0000

Originally published at https://bestlatestapps.com/blog/building-a-real-time-mic-to-speaker-app-on-ios-with-avaudioengine/

Live audio passthrough in Swift: session setup, permissions, feedback, interruptions and route changes, from shipping an app that does exactly this

Most audio tutorials on iOS cover one of two things: playing a file, or recording one. Very few cover the strange in-between case where the microphone and the speaker are both live at the same time, and whatever goes in comes straight back out.

That in-between case is exactly what my app Mic to Speaker – Virtual Mic does. You plug your iPhone into a speaker, or connect over Bluetooth or AirPlay, tap the screen, and your voice comes out of the speaker. I first shipped it in May 2020. Since then it has collected more than 3,400 ratings on the US App Store, and it is still the most popular of the 24 free apps I have published under my developer name.

It looks like a weekend project, and the first working version really is only a few dozen lines of Swift. But the gap between "it works on my desk" and "it works for thousands of people on random speakers, in cars, during phone calls" is where the real work lives.

This is the guide I wish I'd had: permissions, session setup, the engine graph, feedback, interruptions, route changes, background audio and App Store review. I've checked API names and availability against Apple's developer documentation and flag what's new or recently changed.

1. The problem: what "mic to speaker" really needs

On paper the requirement is one sentence: take audio from the microphone and play it through the speaker, right now. In practice that sentence hides three hard requirements.

Low latency. When you hear your own voice back, your brain notices delay quickly. A little sounds "roomy"; more sounds like an echo and makes it hard to keep talking. Every buffer between mic and speaker adds delay, so you want the smallest buffers the device can reliably handle.

The right output route. By default, when an iPhone records and plays at the same time, output goes to the receiver, the small earpiece you hold to your ear for calls. Users expect the loud bottom speaker, their Bluetooth speaker, the car stereo or an AirPlay device. Getting audio to the right place, and keeping it there, is half the job.

No howling feedback. If the mic can hear the speaker, the sound loops and grows into that painful squeal everyone knows from school assemblies. On a phone, with mic and speaker centimetres apart, that's the default outcome unless you plan for it.

I underestimated all three. My first build played through the earpiece. The second howled as soon as I turned the volume up. The third worked beautifully until a phone call came in, then stayed silent until I killed the app. Each section below is, roughly, one of those lessons.

2. Microphone permission, the modern way

Before anything else, you need the user's permission to use the microphone. Two things are required.

First, a purpose string. Add NSMicrophoneUsageDescription to your Info.plist. Apple's documentation is blunt about this: if your app tries to access the microphone without this key, the app exits. Make it specific, for example: "Virtual Mic uses the microphone to play your voice live through your speaker or headphones."

Second, the request itself. In iOS 17, Apple moved record permission from AVAudioSession to a new class, AVAudioApplication. The old AVAudioSession.requestRecordPermission(_:) is deprecated as of iOS 17. The new API has both a completion-handler version and an async version:

class func requestRecordPermission() async -> Bool

If your deployment target is below iOS 17, you need both paths. Mic to Speaker currently supports iOS 16.6 and later, so this is not theoretical for me. Here's a small helper that handles both, plus the "denied" case:

import AVFoundation
import UIKit

@MainActor
enum MicPermissionManager {

    /// Prompts if needed. Returns true if we may record.
    static func request() async -> Bool {
        if #available(iOS 17.0, *) {
            return await AVAudioApplication.requestRecordPermission()
        } else {
            return await withCheckedContinuation { continuation in
                AVAudioSession.sharedInstance().requestRecordPermission { granted in
                    continuation.resume(returning: granted)
                }
            }
        }
    }

    /// The system will not show the prompt again after a denial,
    /// so the only way forward is the Settings app.
    static func openSettings() {
        guard let url = URL(string: UIApplication.openSettingsURLString) else { return }
        UIApplication.shared.open(url)
    }
}

Three practical notes:

Ask at the moment of need, when the user first taps "start", not at launch.
Denied is permanent from the app's side. The system won't prompt again, so show a short explanation and a button that calls openSettings().
Silence is not an error. Apple's docs note that without permission the app captures only silence. The engine starts "fine" and plays nothing, so always check permission first or you'll get confusing "no sound" bug reports.

3. Configuring AVAudioSession

AVAudioSession is how your app tells iOS what kind of audio it does. Get this wrong and nothing else matters.

Category and options

For simultaneous input and output you need the .playAndRecord category. On top of that, the options do most of the heavy lifting:

.defaultToSpeaker: uses the built-in loudspeaker instead of the earpiece when nothing else is connected. This one option fixes "why is it so quiet?"
.allowBluetoothA2DP: lets output go to high-quality Bluetooth speakers and headphones. A2DP is output-only, so the iPhone's mic stays the input, which is usually what a mic-to-speaker app wants.
.allowBluetoothHFP (formerly .allowBluetooth): lets Bluetooth hands-free devices act as input and output. Apple's current docs mark the old .allowBluetooth name deprecated in favour of .allowBluetoothHFP; on an older SDK, use .allowBluetooth. HFP is the "phone call" profile with noticeably lower quality than A2DP, so I enable it only when the user wants a headset's own microphone.
.allowAirPlay: lets output go to AirPlay devices; for .playAndRecord you must set it explicitly. AirPlay buffers over the network, so expect extra latency you can't control.

Mode

The mode tunes the session for a use case:

.default: no special processing; best for music and singing, and required for echo-cancelled input (below).
.voiceChat: tuned for two-way voice. Apple's docs say it optimises EQ for voice, limits routes to voice-suitable ones and automatically applies the Bluetooth HFP option. Great for speech, not for music.
.measurement: minimises system signal processing for the rawest input.

I default to .default and offer voice processing (section 5) as a user choice.

Buffer duration and sample rate

You can ask for a smaller I/O buffer and a particular sample rate:

try session.setPreferredIOBufferDuration(0.005)  // a request, not a guarantee
try session.setPreferredSampleRate(48_000)

These are preferences; the system decides based on hardware and route. After activating, read ioBufferDuration and sampleRate to see what you actually got, and log them. Bluetooth routes often differ from what you asked for.

Echo-cancelled input (iOS 18.2+)

Starting with iOS 18.2, AVAudioSession has three new members:

isEchoCancelledInputAvailable — whether the device and current setup support it
setPrefersEchoCancelledInput(_:) — ask for it
isEchoCancelledInputEnabled — whether the system actually turned it on

Per Apple's docs, it works only with .playAndRecord and .default mode, on certain 2024-or-later iPhones, on the built-in mic and speaker route. It's tuned for a wider range of sounds than voice processing, and can switch off when the route changes (say, a headset is plugged in). Treat it as a bonus on supported hardware, not your feedback strategy.

Temporary speaker override

For a "force loudspeaker" toggle, use overrideOutputAudioPort(_:). It temporarily changes the current route, and the override clears when the route changes again.

Putting it together

import AVFoundation

enum AudioSessionConfigurator {

    static func configure(allowBluetoothMic: Bool = false,
                          preferEchoCancellation: Bool = true) throws {
        let session = AVAudioSession.sharedInstance()

        var options: AVAudioSession.CategoryOptions = [
            .defaultToSpeaker,
            .allowBluetoothA2DP,
            .allowAirPlay
        ]
        if allowBluetoothMic {
            // Use .allowBluetooth if your SDK predates .allowBluetoothHFP.
            options.insert(.allowBluetoothHFP)
        }

        try session.setCategory(.playAndRecord, mode: .default, options: options)
        try session.setPreferredSampleRate(48_000)
        try session.setPreferredIOBufferDuration(0.005)

        if #available(iOS 18.2, *),
           preferEchoCancellation,
           session.isEchoCancelledInputAvailable {
            try session.setPrefersEchoCancelledInput(true)
        }

        try session.setActive(true)

        print("Sample rate: \(session.sampleRate) Hz, "
            + "IO buffer: \(session.ioBufferDuration * 1000) ms, "
            + "route: \(session.currentRoute.outputs.map(\.portType.rawValue))")
    }

    /// Force the built-in speaker (true) or go back to the normal route (false).
    static func setSpeakerOverride(_ on: Bool) throws {
        try AVAudioSession.sharedInstance()
            .overrideOutputAudioPort(on ? .speaker : .none)
    }

    static func deactivate() {
        try? AVAudioSession.sharedInstance()
            .setActive(false, options: .notifyOthersOnDeactivation)
    }
}

4. The AVAudioEngine graph

AVAudioEngine lets you connect audio nodes like guitar pedals. For mic-to-speaker the graph is short:

inputNode → [EQ] → [Reverb] → mainMixerNode → outputNode

inputNode is the current microphone input.
AVAudioUnitEQ gives bass/treble bands and a globalGain boost (documented range −96 to +24 dB).
AVAudioUnitReverb adds room character; wetDryMix runs 0–100%, where 0 is fully dry.
mainMixerNode is created and wired to outputNode the first time you access it. Its outputVolume (0.0–1.0) is a natural master volume.

Format matching

The most common live-audio crash is a format mismatch. My rule: use the input node's own format for every connection carrying the mic signal, and let the mixer convert for the output hardware. Also validate it first: with no input available, the format can come back with a 0 Hz sample rate or zero channels, and connecting with that raises an exception Swift can't catch.

The complete class

Here's a minimal, complete engine class: one object that owns the engine, exposes a few controls and starts and stops cleanly.

import AVFoundation

enum LiveAudioError: Error {
    case noInputAvailable
}

final class LiveAudioEngine {

    private let engine = AVAudioEngine()
    private let eq = AVAudioUnitEQ(numberOfBands: 2)
    private let reverb = AVAudioUnitReverb()

    private(set) var isRunning = false

    // MARK: Controls

    /// Master volume, 0.0 ... 1.0
    var volume: Float {
        get { engine.mainMixerNode.outputVolume }
        set { engine.mainMixerNode.outputVolume = min(max(newValue, 0), 1) }
    }

    /// Extra input gain in dB, clamped to a sane range.
    var boostDB: Float {
        get { eq.globalGain }
        set { eq.globalGain = min(max(newValue, -24), 24) }
    }

    /// Bass and treble in dB.
    var bassDB: Float {
        get { eq.bands[0].gain }
        set { eq.bands[0].gain = min(max(newValue, -12), 12) }
    }
    var trebleDB: Float {
        get { eq.bands[1].gain }
        set { eq.bands[1].gain = min(max(newValue, -12), 12) }
    }

    /// Reverb amount, 0 (off) ... 100
    var reverbMix: Float {
        get { reverb.wetDryMix }
        set { reverb.wetDryMix = min(max(newValue, 0), 100) }
    }

    // MARK: Setup

    init() {
        let bass = eq.bands[0]
        bass.filterType = .lowShelf
        bass.frequency = 150
        bass.gain = 0
        bass.bypass = false

        let treble = eq.bands[1]
        treble.filterType = .highShelf
        treble.frequency = 6_000
        treble.gain = 0
        treble.bypass = false

        reverb.loadFactoryPreset(.mediumRoom)
        reverb.wetDryMix = 0

        engine.attach(eq)
        engine.attach(reverb)
    }

    // MARK: Start / stop

    func start() throws {
        guard !isRunning else { return }

        let input = engine.inputNode
        let format = input.outputFormat(forBus: 0)
        guard format.sampleRate > 0, format.channelCount > 0 else {
            throw LiveAudioError.noInputAvailable
        }

        // Rebuild connections every time: the input format can change
        // between runs (new route, new sample rate).
        engine.disconnectNodeOutput(input)
        engine.disconnectNodeOutput(eq)
        engine.disconnectNodeOutput(reverb)

        engine.connect(input, to: eq, format: format)
        engine.connect(eq, to: reverb, format: format)
        engine.connect(reverb, to: engine.mainMixerNode, format: format)

        engine.prepare()
        try engine.start()
        isRunning = true
    }

    func stop() {
        guard isRunning else { return }
        engine.stop()
        isRunning = false
    }

    /// Voice processing (echo cancellation, AGC, noise suppression).
    /// Must be changed while the engine is stopped.
    func setVoiceProcessing(_ enabled: Bool) throws {
        let wasRunning = isRunning
        stop()
        try engine.inputNode.setVoiceProcessingEnabled(enabled)
        if wasRunning { try start() }
    }

    /// Exposed so the owner can observe engine configuration changes.
    var underlyingEngine: AVAudioEngine { engine }
}

And using it from a view model:

@MainActor
final class MicViewModel: ObservableObject {
    @Published var isLive = false
    @Published var errorMessage: String?

    let audio = LiveAudioEngine()

    func toggle() async {
        if audio.isRunning {
            audio.stop()
            AudioSessionConfigurator.deactivate()
            isLive = false
            return
        }

        guard await MicPermissionManager.request() else {
            errorMessage = "Microphone access is off. Turn it on in Settings."
            return
        }

        do {
            try AudioSessionConfigurator.configure()
            audio.volume = 0.3          // start low, always
            try audio.start()
            isLive = true
        } catch {
            errorMessage = "Couldn't start audio: \(error.localizedDescription)"
        }
    }
}

Note audio.volume = 0.3. Starting quiet is a safety feature, not a nicety.

5. Taming feedback and latency

No single switch removes feedback. There are trade-offs; pick sensible defaults and let the user adjust.

Headphones vs. speaker

The most reliable fix is physics: if the mic can't hear the speaker, there's no loop. With headphones, feedback basically disappears. With a Bluetooth speaker across the room, it's manageable. With the phone's own loudspeaker centimetres from the mic, you're fighting acoustics at any useful volume.

That's why my App Store listing has plain "Safety tips": start with low volume, connect speakers before turning on, increase volume gradually, keep the device away from speakers. They came straight from user reviews and support emails.

Voice processing

setVoiceProcessingEnabled(_:) (on AVAudioIONode, iOS 13+) turns on Apple's voice processing I/O: echo cancellation, automatic gain control and noise suppression, the same family used for calls. isVoiceProcessingAGCEnabled on the input node lets you keep echo cancellation but drop automatic gain.

Change it while the engine is stopped. It reconfigures the I/O unit and the input format can change, which is why LiveAudioEngine rebuilds connections on every start().
It changes the sound. Tuned for speech, it can make singing and instruments sound processed, so I make it a user choice.
It's not magic. It reduces the loop; at high enough volume on the built-in speaker you can still get feedback.

Buffer size trade-offs

Smaller buffers mean lower latency but less time for the audio thread before the next deadline; miss it and you get clicks and dropouts. Larger buffers are smoother, but you hear yourself later. My approach:

Request a small buffer, then read back what you got.
Keep the chain light. Apple's built-in EQ and reverb are fine; heavy custom DSP is where trouble starts.
Never allocate, lock or do file I/O on the real-time thread. In a tap (level meter, recording), copy what you need and hand it off.
Accept that Bluetooth and AirPlay add buffering you can't remove. I tell users plainly that wired gives the lowest delay.

I'm deliberately not quoting millisecond numbers. Latency depends on device, iOS version, route and granted buffer size; the only honest number is the one you measure on your own devices.

6. Interruptions and route changes

This separates a demo from a shipped app. Real phones get calls, alarms, Siri, other apps taking over audio, and headphones unplugged mid-sentence.

Interruptions

When another session (say, a phone call) takes over, the system posts AVAudioSession.interruptionNotification. On .began, the system has already stopped your audio; update state and UI. On .ended, the options may include .shouldResume, a hint that resuming without user input is appropriate.

What's new: Apple's latest docs mark .shouldResume deprecated as of iOS 27 in favour of a newer resumptionRecommendationNotification. It still works on the versions most apps support, so the sample uses it; on the newest SDK, see Apple's "Handling audio interruptions" article.

For a live mic I'm conservative: auto-resume only if the user was live before and the system says resuming is fine. A mic that silently turns itself back on after a call is a privacy surprise.

Route changes

AVAudioSession.routeChangeNotification carries a reason. The critical one is .oldDeviceUnavailable: headphones unplugged or Bluetooth disconnected. Apple's guidance for media apps is to pause; for a live mic it matters even more, because audio jumps to the loudspeaker and can instantly howl. Stop first, let the user decide. For .newDeviceAvailable, keep going but rebuild the graph, since the input format may have changed.

Media services reset

Rarely, the media server restarts and the system posts AVAudioSession.mediaServicesWereResetNotification. Every audio object is now invalid: recreate the engine and reconfigure the session. Separately, AVAudioEngine posts .AVAudioEngineConfigurationChange when the hardware I/O format changes; the engine stops, and you reconnect and restart.

Putting it together

import AVFoundation

@MainActor
final class AudioEventsHandler {

    private var engine: LiveAudioEngine
    private var wasLiveBeforeInterruption = false
    private var observers: [NSObjectProtocol] = []
    var onStateChange: ((Bool) -> Void)?

    init(engine: LiveAudioEngine) {
        self.engine = engine
        observe()
    }

    deinit {
        observers.forEach { NotificationCenter.default.removeObserver($0) }
    }

    private func observe() {
        let center = NotificationCenter.default
        let session = AVAudioSession.sharedInstance()

        observers.append(center.addObserver(
            forName: AVAudioSession.interruptionNotification,
            object: session, queue: .main) { [weak self] note in
                MainActor.assumeIsolated { self?.handleInterruption(note) }
            })

        observers.append(center.addObserver(
            forName: AVAudioSession.routeChangeNotification,
            object: session, queue: .main) { [weak self] note in
                MainActor.assumeIsolated { self?.handleRouteChange(note) }
            })

        observers.append(center.addObserver(
            forName: AVAudioSession.mediaServicesWereResetNotification,
            object: session, queue: .main) { [weak self] _ in
                MainActor.assumeIsolated { self?.handleMediaServicesReset() }
            })

        observers.append(center.addObserver(
            forName: .AVAudioEngineConfigurationChange,
            object: engine.underlyingEngine, queue: .main) { [weak self] _ in
                MainActor.assumeIsolated { self?.restartIfNeeded() }
            })
    }

    private func handleInterruption(_ note: Notification) {
        guard let info = note.userInfo,
              let raw = info[AVAudioSessionInterruptionTypeKey] as? UInt,
              let type = AVAudioSession.InterruptionType(rawValue: raw) else { return }

        switch type {
        case .began:
            wasLiveBeforeInterruption = engine.isRunning
            engine.stop()
            onStateChange?(false)

        case .ended:
            let optionsRaw = info[AVAudioSessionInterruptionOptionKey] as? UInt ?? 0
            let options = AVAudioSession.InterruptionOptions(rawValue: optionsRaw)
            if wasLiveBeforeInterruption && options.contains(.shouldResume) {
                do {
                    try AVAudioSession.sharedInstance().setActive(true)
                    try engine.start()
                    onStateChange?(true)
                } catch {
                    onStateChange?(false)
                }
            }
            wasLiveBeforeInterruption = false

        @unknown default:
            break
        }
    }

    private func handleRouteChange(_ note: Notification) {
        guard let info = note.userInfo,
              let raw = info[AVAudioSessionRouteChangeReasonKey] as? UInt,
              let reason = AVAudioSession.RouteChangeReason(rawValue: raw) else { return }

        switch reason {
        case .oldDeviceUnavailable:
            // Headphones unplugged / Bluetooth gone: stop before we howl.
            engine.stop()
            onStateChange?(false)
        case .newDeviceAvailable, .categoryChange, .override:
            restartIfNeeded()
        default:
            break
        }
    }

    private func handleMediaServicesReset() {
        // Every audio object is invalid now. Rebuild from scratch.
        engine.stop()
        engine = LiveAudioEngine()
        try? AudioSessionConfigurator.configure()
        onStateChange?(false) // let the user start again
    }

    private func restartIfNeeded() {
        guard engine.isRunning else { return }
        engine.stop()
        do {
            try engine.start()   // reconnects with the new input format
        } catch {
            onStateChange?(false)
        }
    }
}

In production you'd also re-register the configuration observer for the new engine after a reset and hand it back to its owner; I've kept the shape minimal.

7. Background audio and App Store notes

Background audio

To keep the mic live with the screen locked, add audio to UIBackgroundModes (Xcode: Background Modes → "Audio, AirPlay, and Picture in Picture"). With an active .playAndRecord session and a running engine, iOS keeps the app alive and shows its microphone-in-use indicator, so users always know.

The flip side: when the user stops, really stop. Stop the engine and deactivate with .notifyOthersOnDeactivation so paused music and podcasts can resume. Holding the session forever drains battery and blocks other audio.

App Store review considerations

A specific purpose string, in the user's words.
Background audio must be real. Declaring the audio mode means actually producing audible content in the background; an idle app holding the session doesn't qualify.
An obvious "live" state: a big on/off control and a clear indicator.
Honest privacy. If you record sessions, say so and say where they're stored, and keep your App Privacy details consistent.
Don't over-promise. If the app can't act as a system-wide mic for a computer or other apps, don't imply it can.

8. Real-world example: Virtual Mic, a.k.a. Mic to Speaker

Everything above comes from building and maintaining my own apps. The main one is on the App Store as Mic to Speaker – Virtual Mic.

At the time of writing, its public App Store listing shows:

Free, in the Music and Utilities categories
First released in May 2020
About 3,400 ratings in the US store, averaging around 4.5 stars
Supports iOS 16.6 and later, on iPhone and iPad

The listing describes the core idea simply: "Use your device as a live microphone! Connect to any speaker and your voice comes out loud and clear." Here's how the listing's features map to the techniques in this article:

What the listing says	What makes it work
"Open the app and tap the screen. That's it!" / "One tap to turn on or off"	Permission request on first tap (section 2), then session + engine start (sections 3–4)
"Connect via Bluetooth, AirPlay, or cable"	`.playAndRecord` with `.defaultToSpeaker`, `.allowBluetoothA2DP`, `.allowAirPlay` (section 3)
"Works in the background" / "Runs even when screen is locked"	`UIBackgroundModes` → `audio`, active session, running engine (section 7)
"Add echo and reverb… Adjust bass and treble. Make your voice louder or softer. Everything updates in real-time"	`AVAudioUnitReverb`, `AVAudioUnitEQ` bands and gain, mixer volume, all live in the graph (section 4)
"Every session is saved automatically… MP3, WAV, M4A"	A tap on the graph feeding a file writer, off the real-time thread (section 5)
"Safety tips: Start with low volume… Avoid feedback by keeping device away from speakers"	The feedback realities in section 5, and starting at a low volume by default

The listing's use cases range from live streaming, karaoke and podcasts to announcements and hearing assistance: amplifying voices in meetings or classes. That last one shapes my defaults most. For someone using your app to hear better, a feedback squeal or surprise route change isn't a glitch; it hurts. Hence "start quiet" and "stop on .oldDeviceUnavailable". It's also why the app still supports iOS 16.6 and carries the pre-iOS 17 permission path from section 2.

The same idea exists on Android as Virtual Mic on Google Play (10K+ downloads at the time of writing). Its listing highlights real-time noise reduction, echo cancellation, reverb and echo effects, an equaliser, input gain and output volume controls, and support for Bluetooth speakers and wired headphones.

On Android the building blocks are AudioRecord (capture) and AudioTrack (playback), or Google's Oboe C++ library over AAudio/OpenSL ES for low-latency work. Same problems (latency, routing, feedback, audio focus instead of interruptions), very different APIs. That's my next article.

9. Before you ship live audio: a 7-point checklist

Permission flow is complete. NSMicrophoneUsageDescription is specific; you ask at the moment of need; iOS 17+ uses AVAudioApplication, older versions fall back to AVAudioSession; denied users get an "Open Settings" path.
The session is configured on purpose. .playAndRecord, .defaultToSpeaker, the Bluetooth options you actually want (A2DP for quality, HFP only when needed), and you log the real sample rate and buffer duration you got.
Formats come from the hardware. You connect using the input node's format, check it's valid, and rebuild connections whenever the route or configuration changes.
Feedback has defaults. Start at low volume, recommend headphones or distance, offer voice processing as a user choice, and use echo-cancelled input only where isEchoCancelledInputAvailable says you can.
Interruptions and route changes are tested on a real device. Phone call, alarm, Siri, unplugging headphones, Bluetooth disconnecting, AirPlay switching: the app ends up in a sensible, visible state every time.
Background behaviour is honest. The audio background mode is declared only because you really play audio in the background, and you deactivate the session with .notifyOthersOnDeactivation when the user stops.
The real-time thread stays clean. No allocations, locks, logging or file I/O inside render callbacks or taps; heavy work is handed off.

Wrapping up

A mic-to-speaker app is a great way to learn iOS audio because there's nowhere to hide: latency is audible, routing mistakes are obvious, and a missed interruption leaves a silent app. AVAudioSession plus AVAudioEngine give you everything you need in a few hundred lines of Swift, as long as you respect the session, follow the hardware's formats and handle real-world events.

If you want to see the result in action, try my apps:

Mic to Speaker – Virtual Mic (iOS): https://apps.apple.com/us/app/mic-to-speaker-virtual-mic/id1514011963
Virtual Mic (Android): https://play.google.com/store/apps/details?id=com.aryansbtloe.microphone.android
All 24 of my free iOS apps: https://apps.apple.com/us/developer/alok-singh/id1508795536
My portfolio: https://bestlatestapps.com/alok/

If you've shipped live audio on iOS and hit a problem I didn't cover, I'd love to hear about it in the comments. Follow me here on Medium for the Android version of this guide.

— Alok Kumar Singh

Originally published at https://bestlatestapps.com/blog/building-a-real-time-mic-to-speaker-app-on-ios-with-avaudioengine/

Text Plus Audio, Fully Offline: Building a Prayer App with AVAudioPlayer and Lock-Screen Controls

Fri, 23 Oct 2026 16:38:02 +0000

Originally published at https://bestlatestapps.com/blog/text-plus-audio-fully-offline/

Bundled recitation audio, verse-by-verse highlighting from a cue sheet, background playback, Now Playing controls and the interruptions you must handle, on an app that still supports iOS 11

Some apps are used in a very particular moment. A prayer app is opened in the morning, often before anything else, frequently by someone older who isn't interested in settings screens, and sometimes somewhere with no signal at all. It has to open instantly, play immediately and never ask for an account.

Hanuman Chalisa : Offline was the second app I published, in April 2020, one day after my Bible app. The Hanuman Chalisa is a devotional hymn to Lord Hanuman, attributed to the poet-saint Tulsidas. The app does two things: it shows the Chalisa in English-language script for reading, and it plays an audio recitation, all without an internet connection.

That combination, text plus audio that stay in step, is a pattern you'll find in language-learning apps, audiobooks with text, karaoke lyrics and guided meditations. This article walks through how I'd build it on iOS: bundling text and audio, a cue sheet for verse timing, AVAudioPlayer with a proper .playback session, highlighting the current verse, lock-screen controls through MPNowPlayingInfoCenter and MPRemoteCommandCenter, and the interruptions and route changes a real device throws at you. The app supports iOS 11 and later, so the code is UIKit-first and stays on long-established APIs. I'm describing the approach I'd use and recommend, not the app's exact source.

1. What "offline text plus audio" requires

Three requirements shape the design:

Zero network, zero setup. Everything ships in the app bundle. First launch is the same as the hundredth.
Audio and text are one experience. When the recitation reaches a verse, that verse is highlighted and scrolled into view. When the user taps a verse, the audio jumps there.
It behaves like a real audio app. It keeps playing with the screen locked, shows controls on the Lock Screen and in Control Center, pauses for a phone call, and stops when headphones are pulled out.

The first two are about your data model. The third is about respecting the system.

2. Bundle the content with a cue sheet

I keep the text and the timing together in one small JSON file in the bundle, next to the audio file:

{
  "title": "Hanuman Chalisa",
  "audio": "chalisa.m4a",
  "verses": [
    { "id": 1, "kind": "doha",  "start": 0.0,  "text": "…" },
    { "id": 2, "kind": "doha",  "start": 14.2, "text": "…" },
    { "id": 3, "kind": "chaupai", "start": 27.8, "text": "…" }
  ]
}

Each verse carries the time, in seconds, at which its recitation starts. That's the whole cue sheet. You produce it once, by listening with a waveform editor open, and it never changes unless the recording does.

import Foundation

struct Verse: Decodable {
    let id: Int
    let kind: String
    let start: TimeInterval
    let text: String
}

struct Recitation: Decodable {
    let title: String
    let audio: String
    let verses: [Verse]

    static func loadBundled(named name: String = "chalisa") throws -> Recitation {
        guard let url = Bundle.main.url(forResource: name, withExtension: "json") else {
            throw CocoaError(.fileNoSuchFile)
        }
        let recitation = try JSONDecoder().decode(Recitation.self, from: Data(contentsOf: url))
        // Cue sheets must be sorted; fail loudly in debug if someone edits one by hand.
        assert(zip(recitation.verses, recitation.verses.dropFirst()).allSatisfy { $0.start < $1.start })
        return recitation
    }

    /// Index of the verse being recited at `time` (binary search over start times).
    func verseIndex(at time: TimeInterval) -> Int {
        var low = 0, high = verses.count - 1, answer = 0
        while low <= high {
            let mid = (low + high) / 2
            if verses[mid].start <= time { answer = mid; low = mid + 1 } else { high = mid - 1 }
        }
        return answer
    }
}

A few practical notes:

AAC (.m4a) is the right format for a bundled spoken or sung recording: small, and decoded in hardware.
Keep the text as Unicode text, not images. It scales with Dynamic Type, it works with VoiceOver, and you can offer other scripts later from the same structure.
Binary search is overkill for 40-odd verses, but it costs nothing, and the same code works for a 300-chapter audiobook.

3. AVAudioPlayer and a session that keeps playing

For a single bundled file, AVAudioPlayer (available since iOS 2.2) is all you need. There's no engine graph and nothing to stream. What matters is the audio session category:

import AVFoundation

final class RecitationPlayer: NSObject, AVAudioPlayerDelegate {
    private(set) var player: AVAudioPlayer?
    var onFinish: (() -> Void)?

    func load(_ recitation: Recitation) throws {
        guard let url = Bundle.main.url(forResource: recitation.audio, withExtension: nil) else {
            throw CocoaError(.fileNoSuchFile)
        }
        let player = try AVAudioPlayer(contentsOf: url)
        player.delegate = self
        player.enableRate = true          // must be set before prepareToPlay() to allow rate changes
        player.prepareToPlay()
        self.player = player
    }

    func play() throws {
        let session = AVAudioSession.sharedInstance()
        // .playback: audio is central to the app; it plays with the Silent switch on
        // and, with the "audio" background mode, keeps going when the screen locks.
        try session.setCategory(.playback, mode: .spokenAudio, options: [])
        try session.setActive(true)
        player?.play()
    }

    func pause() { player?.pause() }

    func seek(to time: TimeInterval) {
        guard let player else { return }
        player.currentTime = max(0, min(time, player.duration))
    }

    func audioPlayerDidFinishPlaying(_ player: AVAudioPlayer, successfully flag: Bool) {
        try? AVAudioSession.sharedInstance().setActive(false, options: .notifyOthersOnDeactivation)
        onFinish?()
    }
}

Three details that matter more than they look:

.playback, not the default. The default category (.soloAmbient) is silenced by the ring/silent switch and stops when the screen locks. For a prayer app, both are bugs.
Add the background mode. In Xcode, enable Background Modes → "Audio, AirPlay, and Picture in Picture" (audio in UIBackgroundModes). Without it, .playback alone won't keep you alive after the screen locks.
The .spokenAudio mode (iOS 9+) is meant for continuous spoken audio. Per Apple's docs, your audio pauses rather than ducks when another app plays a short spoken prompt, such as a navigation instruction, and you can resume afterwards. A sung recitation sits somewhere between speech and music, so .default is a reasonable choice too. setCategory(_:mode:options:) is iOS 10+, so it's safe on an iOS 11 target.

4. Highlighting the current verse

To keep the text in step with the audio, poll currentTime a few times a second and move the highlight when the verse index changes. On iOS 11 I use a CADisplayLink (or a plain Timer) rather than anything newer:

import UIKit

final class RecitationViewController: UITableViewController {
    private let recitation: Recitation
    private let audio = RecitationPlayer()
    private var link: CADisplayLink?
    private var currentIndex = -1

    init(recitation: Recitation) {
        self.recitation = recitation
        super.init(style: .plain)
    }
    required init?(coder: NSCoder) { fatalError("init(coder:) is not supported") }

    override func viewDidLoad() {
        super.viewDidLoad()
        tableView.register(UITableViewCell.self, forCellReuseIdentifier: "verse")
        tableView.rowHeight = UITableView.automaticDimension
        try? audio.load(recitation)
    }

    func startPlayback() {
        try? audio.play()
        link = CADisplayLink(target: self, selector: #selector(tick))
        link?.preferredFramesPerSecond = 10     // plenty for text; saves battery
        link?.add(to: .main, forMode: .common)
    }

    func stopUpdates() {
        link?.invalidate(); link = nil
    }

    @objc private func tick() {
        guard let time = audio.player?.currentTime else { return }
        let index = recitation.verseIndex(at: time)
        guard index != currentIndex else { return }
        let previous = currentIndex
        currentIndex = index
        let rows = [previous, index].filter { $0 >= 0 }.map { IndexPath(row: $0, section: 0) }
        tableView.reloadRows(at: rows, with: .fade)
        tableView.scrollToRow(at: IndexPath(row: index, section: 0), at: .middle, animated: true)
    }

    // Tap a verse to jump the audio there.
    override func tableView(_ tableView: UITableView, didSelectRowAt indexPath: IndexPath) {
        audio.seek(to: recitation.verses[indexPath.row].start)
        tableView.deselectRow(at: indexPath, animated: true)
    }

    override func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
        recitation.verses.count
    }

    override func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {
        let cell = tableView.dequeueReusableCell(withIdentifier: "verse", for: indexPath)
        let isCurrent = indexPath.row == currentIndex
        cell.textLabel?.text = recitation.verses[indexPath.row].text
        cell.textLabel?.numberOfLines = 0
        cell.textLabel?.font = UIFont.preferredFont(forTextStyle: isCurrent ? .headline : .body)
        cell.textLabel?.adjustsFontForContentSizeCategory = true
        cell.backgroundColor = isCurrent ? UIColor.orange.withAlphaComponent(0.15) : .clear
        return cell
    }
}

Two accessibility points that matter especially for this audience:

Dynamic Type everywhere (preferredFont(forTextStyle:) with adjustsFontForContentSizeCategory). Many people reading a prayer app use larger text sizes.
Don't fight the user's scrolling. If they've scrolled away to read ahead, pause auto-scrolling for a few seconds. Detect it in scrollViewWillBeginDragging(_:).

Stop the display link when the view disappears or playback ends. A display link running in the background is wasted battery.

5. Lock Screen and Control Center

Once you're a real .playback app, people expect to control it from the Lock Screen, Control Center, headphones and the car. That's two APIs: MPNowPlayingInfoCenter (iOS 5+) for what's playing and MPRemoteCommandCenter (iOS 7.1+) for what the buttons do.

import MediaPlayer
import UIKit

final class NowPlayingController {
    private weak var audio: RecitationPlayer?

    init(audio: RecitationPlayer, title: String, artwork: UIImage?) {
        self.audio = audio
        configureCommands()
        var info: [String: Any] = [
            MPMediaItemPropertyTitle: title,
            MPMediaItemPropertyArtist: "Recitation",
            MPMediaItemPropertyPlaybackDuration: audio.player?.duration ?? 0
        ]
        if let artwork {
            info[MPMediaItemPropertyArtwork] =
                MPMediaItemArtwork(boundsSize: artwork.size) { _ in artwork }
        }
        MPNowPlayingInfoCenter.default().nowPlayingInfo = info
    }

    /// Call on play, pause and seek, not every frame. The system extrapolates
    /// elapsed time from the rate you give it.
    func update() {
        guard let player = audio?.player else { return }
        var info = MPNowPlayingInfoCenter.default().nowPlayingInfo ?? [:]
        info[MPNowPlayingInfoPropertyElapsedPlaybackTime] = player.currentTime
        info[MPNowPlayingInfoPropertyPlaybackRate] = player.isPlaying ? 1.0 : 0.0
        MPNowPlayingInfoCenter.default().nowPlayingInfo = info
    }

    private func configureCommands() {
        let center = MPRemoteCommandCenter.shared()

        center.playCommand.addTarget { [weak self] _ in
            try? self?.audio?.play(); self?.update(); return .success
        }
        center.pauseCommand.addTarget { [weak self] _ in
            self?.audio?.pause(); self?.update(); return .success
        }
        center.togglePlayPauseCommand.addTarget { [weak self] _ in
            guard let player = self?.audio?.player else { return .commandFailed }
            if player.isPlaying { self?.audio?.pause() } else { try? self?.audio?.play() }
            self?.update(); return .success
        }
        center.changePlaybackPositionCommand.addTarget { [weak self] event in
            guard let e = event as? MPChangePlaybackPositionCommandEvent else { return .commandFailed }
            self?.audio?.seek(to: e.positionTime); self?.update(); return .success
        }
        // Commands you don't support should be disabled, so the UI doesn't show dead buttons.
        center.nextTrackCommand.isEnabled = false
        center.previousTrackCommand.isEnabled = false
    }
}

MPMediaItemArtwork(boundsSize:requestHandler:) is iOS 10+, so it's fine on iOS 11. The scrubber on the Lock Screen needs changePlaybackPositionCommand (iOS 9.1+); without it, the scrubber is display-only.

Don't update Now Playing on every tick. Set elapsed time and rate on play, pause and seek, and the system animates the progress itself.

6. Interruptions and route changes

Real phones ring. Here are the two notifications every audio app must handle:

import AVFoundation

final class AudioInterruptionHandler {
    private weak var audio: RecitationPlayer?
    private var tokens: [NSObjectProtocol] = []
    private var wasPlaying = false
    var onChange: (() -> Void)?

    init(audio: RecitationPlayer) {
        self.audio = audio
        let center = NotificationCenter.default
        let session = AVAudioSession.sharedInstance()

        tokens.append(center.addObserver(forName: AVAudioSession.interruptionNotification,
                                         object: session, queue: .main) { [weak self] note in
            self?.handleInterruption(note)
        })
        tokens.append(center.addObserver(forName: AVAudioSession.routeChangeNotification,
                                         object: session, queue: .main) { [weak self] note in
            self?.handleRouteChange(note)
        })
    }

    deinit { tokens.forEach(NotificationCenter.default.removeObserver) }

    private func handleInterruption(_ note: Notification) {
        guard let raw = note.userInfo?[AVAudioSessionInterruptionTypeKey] as? UInt,
              let type = AVAudioSession.InterruptionType(rawValue: raw) else { return }
        switch type {
        case .began:
            wasPlaying = audio?.player?.isPlaying ?? false   // the system has already paused us
        case .ended:
            let optionsRaw = note.userInfo?[AVAudioSessionInterruptionOptionKey] as? UInt ?? 0
            if wasPlaying, AVAudioSession.InterruptionOptions(rawValue: optionsRaw).contains(.shouldResume) {
                try? audio?.play()
            }
            wasPlaying = false
        @unknown default: break
        }
        onChange?()
    }

    private func handleRouteChange(_ note: Notification) {
        guard let raw = note.userInfo?[AVAudioSessionRouteChangeReasonKey] as? UInt,
              let reason = AVAudioSession.RouteChangeReason(rawValue: raw) else { return }
        // Headphones unplugged / Bluetooth gone: pause, as Apple recommends for media apps,
        // instead of suddenly reciting out loud on the speaker.
        if reason == .oldDeviceUnavailable { audio?.pause(); onChange?() }
    }
}

Two version notes. The Swift names AVAudioSession.interruptionNotification and routeChangeNotification are the modern spellings; on the very oldest SDKs they were NSNotification.Name.AVAudioSessionInterruption and AVAudioSessionRouteChange. And Apple's current docs mark .shouldResume as deprecated in iOS 27 in favour of a newer resumption-recommendation mechanism. It still works on every version this app supports, so I've kept it here.

7. Real-world example: Hanuman Chalisa : Offline

This is the approach I'd use for Hanuman Chalisa : Offline, and for any text-plus-audio app.

At the time of writing, its App Store listing shows:

Free, in the Book and Music categories
First released in April 2020
22 ratings in the US store, averaging about 4.8 stars, the highest average among my apps with ten or more ratings
Supports iOS 11.0 and later, on iPhone, iPad and iPod touch

The listing describes the content simply: "This version has both Chalisa in English Language Script (for reading) and Audio." Here's how that maps to this article:

What the listing says	Where it's covered
"Hanuman Chalisa : Offline"	Text, cue sheet and audio all bundled; no network code at all (section 2)
"Chalisa in English Language Script (for reading)"	Unicode text with Dynamic Type in a table view (section 4)
"…and Audio"	`AVAudioPlayer` with a `.playback` session and background mode (section 3)
"The iPhone, iPod & iPad version"	UIKit-first code on long-standing APIs, iOS 11 and up (throughout)

I later built a much larger devotional app, Mandir Darshan, with live temple streams, the Bhagavad Gita and a chalisa collection. That one's a very different technical story, built on live streaming and picture-in-picture, and I'm covering it in its own article.

8. Before you ship a text-plus-audio app: a checklist

It works in Airplane Mode from a fresh install: text, audio and highlighting.
The category is .playback and the audio background mode is enabled, so audio survives the Silent switch and the Lock Screen.
The cue sheet is validated. Start times are sorted, and every verse has one. Assert it in debug builds.
Tapping a verse seeks the audio, and the highlight follows the audio, never the other way round.
Lock-screen controls work, including the scrubber, and unsupported commands are disabled.
A phone call pauses playback, and it resumes only if it was playing and the system says it may.
Unplugging headphones pauses playback.
Text respects Dynamic Type and VoiceOver. Test at the largest accessibility sizes.
The session is deactivated when playback finishes, with .notifyOthersOnDeactivation, so other audio can resume.

Wrapping up

A text-plus-audio app looks simple, and it can be, as long as you treat the audio session with respect. Bundle everything, describe timing with a tiny cue sheet, let AVAudioPlayer play, keep the highlight driven by currentTime, and give the system what it needs for the Lock Screen and for interruptions. It's well-established API, which is exactly why it still runs on iOS 11.

Try the app, or explore my other work:

Hanuman Chalisa : Offline (iOS): https://apps.apple.com/us/app/hanuman-chalisa-offline/id1509206340
All of my free iOS apps: https://apps.apple.com/us/developer/alok-singh/id1508795536
My portfolio: https://bestlatestapps.com/alok/

Have you built synchronised text and audio for lyrics, language lessons or audiobooks? I'd love to hear how you produced your timings. Tell me in the comments.

— Alok Kumar Singh

Originally published at https://bestlatestapps.com/blog/text-plus-audio-fully-offline/

A Practical QA Checklist for Browser-Based Song Search

Fri, 23 Oct 2026 07:38:41 +0000

Disclosure: I work on Hum to Search. This is a practical input-quality and review checklist, not an accuracy benchmark or a description of our private recognition backend.

When a user remembers a melody but not a title, a search box is not always enough. A browser-based song search needs a different starting point: determine what kind of evidence the user actually has.

Our song finder presents separate "Hum or sing" and "Identify a recording" modes. That distinction is useful when designing a test plan. A person humming a chorus and a recording of the original song are different inputs; a good interface should not make the user guess which path to choose.

Start with an input checklist

Before searching, answer three questions:

Is this a hummed melody, singing, or a recording of music playing elsewhere?
Is the useful section audible, or is most of the clip silence or conversation?
Can the person record another attempt without replaying the same weak sample?

For humming, the current input screen suggests 10-20 seconds in a quiet place and trying the chorus without background music. Treat that as capture guidance, not a promise of a match. A clear, continuous phrase is a better test input than several disconnected fragments.

For a recording, keep a short section where the music itself is audible. Do not upload unrelated conversations or other people's private audio just to identify background music.

Separate capture problems from recognition problems

A failed result can mean several things. The microphone may not have captured sound, the recording may be unclear, or the supplied evidence may not produce a useful candidate. These should be different states in a product test report.

Here is a small QA matrix a developer can adapt:

Test input	What to inspect	Useful next action
Microphone access declined	Is the reason visible?	Explain how to retry or choose upload
Silent recording	Is there usable audio?	Record again before another search
Hummed chorus	Correct input mode and continuous melody	Compare a different clear phrase
Music with speech over it	Whether music is actually audible	Choose a cleaner section
Ambiguous candidate	Title, artist and familiar melody	Listen and verify rather than accept blindly

This is a proposed test matrix. It does not claim that every browser or every case has passed testing on our site.

Keep a lightweight test log

For a reproducible evaluation, write down the conditions rather than just saying "it worked" or "it failed". A simple CSV is enough:

case_id,input_mode,clip_seconds,environment,expected_song,result,review_note
hum-01,hum,15,quiet,known_test_track,pending,compare chorus
record-01,recording,12,background_speech,known_test_track,pending,try cleaner section

These are example rows, not measured results. Use audio you own or have permission to test. Keep identifying information out of shared logs, and set your own retention policy before keeping recordings.

Change one variable at a time. If the first attempt is noisy, improve the recording before changing the melody, device and duration together. Otherwise you will not know which change helped.

A candidate still needs a human check

A returned title is a candidate to investigate, not a reason to stop thinking. Listen to a legitimate preview and compare the part you remember. Covers, live recordings and similar melodies can make a result feel familiar without being the exact track you wanted.

For product design, "no useful match" deserves its own helpful recovery path. Suggest a clearer phrase or a different input mode instead of repeating a generic error. If the service uses credits, show the current cost before the user submits; avoid promising unlimited retries unless the actual plan supports them.

Try the workflow

Hum to Search is our browser-based starting point for this workflow. Choose the appropriate input mode, provide a clear sample, and review the candidate instead of assuming recognition is certain. Check the live interface and pricing page for current usage conditions.

Website: https://humtosearch.app/

For developers building an audio interface, the practical takeaway is simple: make input type explicit, make failures distinguishable, and make the next attempt easier to understand.

Three audio checks worth running before you ship any upload feature

Fri, 16 Oct 2026 13:08:03 +0000

Most "the audio is broken" reports are not DSP bugs. They are format and metadata bugs that surface weeks later, after the feature has already shipped. These three checks are cheap and catch the majority of them.

1. Read the header, not the extension

A .wav can be 8-bit PCM, 24-bit float, or something a recorder invented. If you branch on the file extension you will eventually hand a decoder something it silently mangles. Parse the actual header first, then decide what to do with it.

2. Sample rate mismatches are silent

Drop a 44.1 kHz source into a 48 kHz graph and nothing throws. It just plays back at the wrong speed and pitch. Normalise on ingest, then assert the rate at every stage boundary. An assertion costs nothing; a support ticket does not.

3. Peak level is not loudness

A track can peak at 0 dBFS and still sound quiet next to its neighbours. If your UI says "loud" or "quiet", you are measuring loudness (LUFS), not peaks. Pick one metric, name it in the UI, and stop guessing which one you meant.

None of this needs a DSP background. It needs one explicit place in the pipeline where the invariant is written down and checked.

These three came straight out of debugging the upload path on Creatune AI Music Generator, which I work on — so treat the recommendation with the usual scepticism.

Kempo: Real-Time Audio Time-Stretching, Pitch-Shifting, and Formant Control in Pure KMP

Wed, 14 Oct 2026 07:01:23 +0000

Zero-allocation streaming, phase-locked clarity, and multichannel spatial coherence across JVM, Android, iOS, and WebAssembly.

Whether you are building a podcast player with variable playback speed, a music practice app that slows down a complex guitar solo without dropping the tuning, a real-time voice modulator, or a spatial audio engine for a game, you eventually hit the same fundamental challenge: decoupling time from pitch.

In raw digital audio, playback speed and pitch are locked together. Play a buffer twice as fast, and the pitch jumps up a full octave. Slow it down, and everything drags into a muddy baritone.

Breaking that link—while keeping drums punchy, vocals natural, and the stereo image intact—requires a dedicated Digital Signal Processing (DSP) engine.

Meet Kempo (io.github.numq:kempo): an open-source, pure Kotlin Multiplatform audio time-stretching and pitch-shifting library built for studio-grade acoustic transparency and sub-millisecond real-time streaming across JVM, Android, iOS, and WebAssembly (Wasm/JS).

What Kempo Brings to Your Audio Stack

Kempo is designed to be drop-in simple for high-level application code while exposing full, granular control for low-level audio threads:

Independent Time, Pitch, and Formant Control: Speed up or slow down audio (0.25x to 3.0x and beyond), transpose musical pitch in semitones, and shift vocal timbre independently.
Zero-Allocation Real-Time Streaming: Once configured, the block-by-block processing loop allocates 0 bytes on the heap, eliminating Garbage Collection pauses on high-priority audio threads.
Vertical Harmonic Phase-Locking: Keeps transients crisp and eliminates the hollow, "underwater" phasing artifacts common in basic time-stretchers.
Multichannel & 5.1 Surround Coherence: Preserves exact inter-channel phase relationships across stereo and surround tracks so the soundstage never collapses.
Sample-Accurate Offline Rendering: Built-in pre-roll warm-up, algorithmic latency compensation, and automatic overlap-add tail flushing.
Built-In PCM-16 Serialization: Convert raw little-endian or big-endian WAV/PCM byte arrays directly into normalized floating-point tracks and back.

How Kempo Keeps Audio Crisp (Without the Artifacts)

If you have ever used a naive time-stretching algorithm, you probably recognize three classic audio defects: smeared drum hits, "chipmunk" vocals, and a collapsing stereo field. Kempo solves all three at the engine level.

1. Eliminating Transient Smearing with Vertical Phase-Locking

Standard phase vocoders split audio into hundreds of narrow frequency bins and advance the phase of each bin over time. Because each bin is updated in isolation, the natural harmonic alignment between a fundamental note and its overtones quickly drifts apart. Percussive attacks lose their snap, and instruments sound phasey and diffuse.

Kempo continuously analyzes the smoothed spectral energy contour of each frame to locate prominent spectral peaks. Instead of letting neighboring bins drift, Kempo performs vertical phase-locking—anchoring the phase trajectories of surrounding frequency bins to their dominant harmonic peaks. Harmonics move together as a unified acoustic event, keeping transients tight.

When stretching audio to extreme lengths (beyond 2.0x), Kempo automatically transitions into controlled stochastic phase dispersion, preventing the metallic ringing that usually plagues ultra-slow playback.

2. Fixing the "Chipmunk" Effect with Formant Decoupling

When you pitch a human voice up by +5 semitones, you aren't just raising the pitch of the vocal cords—you are also shrinking the perceived size of the singer's vocal tract. The result is the notorious "chipmunk" effect (or the "monster" effect when pitching down).

In Kempo, pitch and vocal timbre (formants) are decoupled. Kempo extracts the spectral envelope of the signal and allows you to shift or lock formants independently of the musical transposition:

Natural Pitch Shifting: Enable formant compensation (compensatePitch = true) to keep a singer's vocal tract resonance anchored while changing their melody.
Creative Voice Design: Shift formants up or down while keeping pitch at 0 st to transform the character and age of a voice without changing the note being sung.

3. Preserving Stereo Width and 5.1 Surround Imaging

Our ears localize sound in space by detecting tiny timing and phase differences between the left and right channels. If an audio processor stretches the Left and Right channels independently, their relative phases drift apart—causing the stereo image to wobble and center-panned vocals to hollow out when downmixed.

Kempo processes multichannel audio coherently. At every frequency band, it identifies the dominant channel and locks the Inter-channel Phase Differences (IPD) of all remaining channels to that reference. Whether you are processing a stereo master or a discrete 6-channel 5.1 surround mix, panning stays locked and center dialogue remains phase-aligned.

Getting Started

Kempo is published on Maven Central. Add the dependency to your build.gradle.kts:

kotlin {
    sourceSets {
        commonMain.dependencies {
            implementation("io.github.numq:kempo:1.0.0")
        }
    }
}

Or for a single-platform JVM/Android module:

dependencies {
    implementation("io.github.numq:kempo:1.0.0")
}

1. The One-Liner: Offline Track Processing

If you have raw 16-bit PCM bytes (from a WAV file, network stream, or microphone recording) and want to stretch or transpose them in a background coroutine, PcmConverter and AudioTrack handle the entire pipeline in a few lines:

import io.github.numq.kempo.PcmConverter

// 1. Decode interleaved 16-bit PCM bytes into a normalized [-1.0f, 1.0f] AudioTrack
val track = PcmConverter.fromPcm16(
    bytes = rawWavBytes,
    channels = 2,
    sampleRate = 44100f,
    isLittleEndian = true
)

// 2. Speed up by 25%, pitch up by 3 semitones, keep natural formants
val stretchedTrack = track.stretch(
    timeRatio = 1.25f,
    pitchSemitones = 3.0f,
    formantSemitones = 0.0f
)

// 3. Convert back to 16-bit PCM bytes for playback or file export
val outputBytes = PcmConverter.toPcm16(stretchedTrack, isLittleEndian = true)

2. Zero-Allocation Real-Time Streaming

Real-time audio callbacks operate under strict deadlines. Allocating arrays or objects inside the render loop will eventually trigger the garbage collector and cause audible clicks.

KempoStretch pre-allocates all ring buffers, FFT plans, twiddle factors, and spectral workspaces upfront during presetDefault() or configure(). Inside your audio callback, process() mutates pre-allocated buffers in place:

import io.github.numq.kempo.KempoStretch
import kotlin.math.max

class LiveAudioProcessor(
    channels: Int = 2,
    sampleRate: Float = 48000f
) {
    private val stretch = KempoStretch().apply {
        presetDefault(channels, sampleRate)
    }

    val outBlockSize: Int = stretch.defaultInterval
    private val outBuffer = Array(channels) { FloatArray(outBlockSize) }

    fun updateParameters(pitchSemitones: Float, formantSemitones: Float) {
        stretch.setTransposeSemitones(pitchSemitones)
        stretch.setFormantSemitones(formantSemitones, compensatePitch = true)
    }

    // Safe for high-priority audio threads: 0 B heap allocations
    fun processFrame(
        inputChannels: Array<FloatArray>,
        playbackSpeed: Float
    ): Array<FloatArray> {
        val inSamples = max(1, (outBlockSize * playbackSpeed).toInt())
        stretch.process(
            inputs = inputChannels,
            inputSamples = inSamples,
            outputs = outBuffer,
            outputSamples = outBlockSize
        )
        return outBuffer
    }
}

Need to support interactive scrubbing on a timeline without clicks or phase resets? Use stretch.seek() before resuming playback to pre-seed the internal phase accumulators with the audio frames right before the new cursor position:

val seekPreRoll = minOf(stretch.seekLength(), targetSampleIndex)
stretch.seek(
    inputs = track.channels,
    inputSamples = seekPreRoll,
    playbackRate = 1.0,
    inputOffset = (targetSampleIndex - seekPreRoll).coerceAtLeast(0)
)

3. Sample-Accurate Offline Rendering with `exact()`

When rendering an audio clip for a DAW timeline or a game asset pipeline, you often need the output buffer to match an exact target sample count—without leading silence from the STFT window latency or cut-off reverb tails at the end.

KempoStretch.exact() automates the entire offline lifecycle: it calculates the required pre-roll (outputSeekLength), warms up the vocoder state (outputSeek), processes the body of the track, and flushes the final synthesis tails (flush):

import io.github.numq.kempo.KempoStretch
import kotlin.math.roundToInt

fun renderExactClip(
    input: Array<FloatArray>,
    sampleRate: Float,
    timeRatio: Float,
    pitchSemitones: Float
): Array<FloatArray> {
    val channels = input.size
    val inSamples = input[0].size
    val outSamples = (inSamples / timeRatio).roundToInt().coerceAtLeast(1)
    val output = Array(channels) { FloatArray(outSamples) }

    val stretch = KempoStretch().apply {
        presetDefault(channels, sampleRate)
        setTransposeSemitones(pitchSemitones)
    }

    stretch.exact(
        inputs = input,
        inputSamples = inSamples,
        outputs = output,
        outputSamples = outSamples
    )
    return output
}

4. Custom Frequency Mapping & Tonality Limits

For experimental sound design, spectral effects, or specialized speech conditioning, Kempo lets you bypass standard semitone math entirely:

Tonality Limits: When transposing speech or vocals up, multiplying high-frequency sibilance ("s" and "sh" sounds) can make them sound lispy or artificial. Passing a tonalityLimit to setTransposeSemitones(semitones, tonalityLimit) transitions the upper spectrum from multiplicative harmonic scaling to a clean linear shift.
Custom Transfer Functions: Pass any lambda to setFreqMap to warp normalized frequencies [0.0f, 0.5f] non-linearly:

stretch.setFreqMap { normalizedFreq ->
    // Custom non-linear spectral warping
    (normalizedFreq * 1.15f).coerceIn(0f, 0.5f)
}

Engineered for Speed in Pure Kotlin

Writing high-throughput DSP in pure Kotlin without falling back to platform-specific C/C++ libraries requires careful attention to memory layout and CPU cache locality:

Flat Power-of-Two Ring Buffers (MultiChannelBuffer): Instead of nested object arrays or modulo arithmetic, Kempo backs all channels with a single flat FloatArray. Wrap-around indices are resolved via bitwise masks ((head + offset) and mask), and overlap-add accumulation uses 4x unrolled loops.
In-Place Mixed-Radix FFT (FastFft & ModifiedRealFFT): Supports radix-2, radix-3, radix-4, and composite window sizes with precomputed twiddle tables. Real signals are evaluated on a half-bin shifted grid (k + 0.5) / N, avoiding DC and Nyquist edge singularities and cutting transform overhead in half.
Split Computation Mode (split = true): In real-time audio, doing a full STFT analysis and synthesis every N samples can cause periodic CPU spikes. Enabling split = true in configure() or presetCheaper() slices the spectral workload into small incremental steps distributed evenly across every sample frame.
Automatic Silence Bypass: When input energy drops below the noise floor, Kempo automatically bypasses heavy spectral math while keeping ring buffers synchronized.

In JMH benchmarks on standard desktop JVM hardware (44.1 kHz audio), Kempo processes a real-time streaming block in ~220 μs (mono) and ~484 μs (stereo), and renders offline audio at over 128x real-time speed for mono and ~70x real-time for stereo.

Try It Live: The Compose Desktop Studio App

Want to hear how Kempo handles your own audio files before writing a line of code?

The repository includes an interactive Compose Multiplatform Desktop Studio application (:example). Drop any .wav file onto the window to inspect its waveform, scrub through the timeline in real time, and tweak live parameters on the fly:

Playback Speed (0.25x – 3.00x) with instant presets (0.50x, 0.75x, 1.25x, 1.50x, 2.00x)
Pitch Shift (-12 st to +12 st) in semitone steps
Formant Shift (-12 st to +12 st) for real-time vocal timbre shaping

Explore the Code

Check out the full source code, architecture docs, and the runnable CLI attack simulation on GitHub.

How AI Music Generators Turn a Text Prompt Into Audio

Thu, 24 Sep 2026 23:09:20 +0000

If you have only used AI music tools from the outside, the pipeline underneath is a nice piece of engineering that borrows heavily from the language models most of us already work with. This is a short tour of how text becomes a finished track, written for developers who want to know what is actually running.

Step One: Turn Audio Into Tokens

Raw audio is a continuous waveform with tens of thousands of samples per second, far too much for a language style model to predict directly. Neural codecs like Meta's EnCodec and Google's SoundStream compress it into sequences of discrete tokens, so a minute of audio becomes a few thousand tokens instead of millions of samples. Each token captures both what is playing and how it sounds, and that compression is what makes generation computationally feasible.

Step Two: Generate the Tokens

Two architectures dominate. Autoregressive transformers, the approach behind Suno and Meta's MusicGen, predict one audio token at a time from everything that came before it, exactly like next word prediction in a chat model. They tend to be faster and better at holding long range structure, like a verse coming back after the chorus. Diffusion models, used by Stability AI's Stable Audio, start from noise and refine it over many steps, which often gives cleaner and more detailed textures.

Step Three: Condition on the Prompt

To make "upbeat jazz piano trio with walking bass" mean something, the system uses a joint embedding model such as CLAP or MuLan. These are trained so that a text description and the matching audio land close together in the same vector space, and during generation the text embedding steers the model toward audio that fits. The final tokens are decoded back into a WAV or MP3, sometimes with a light mastering pass on top.

Running It Yourself

MusicGen and Stable Audio Open are both open models, so you can run the whole pipeline on your own GPU with no per track license terms. The hosted tools differ mostly in audio quality, vocals and licensing, and this breakdown of AI music generators covers how Suno, Udio, SOUNDRAW and the rest compare on those points.

The Takeaway

Under the hood, an AI music generator is a codec, a sequence model and a text encoder wired together, the same building blocks as the language models we ship every day. If you have worked with embeddings and next token prediction, you already understand most of it.

Why Voice Practice Improves When Recording and Evaluation Are Separate

Sun, 30 Aug 2026 13:21:18 +0000

A browser voice-practice tool looks simple from the outside: show a line, turn on the microphone, record a take, and play it back. In practice, the most important design decision is not the recorder API. It is deciding when the performer should perform and when they should evaluate.

Trying to do both at once creates unnecessary cognitive load. A user watches the script, remembers the character, controls timing, listens to their own delivery, and judges the result simultaneously. The result is usually a cautious performance and vague feedback: "something felt off."

A better workflow separates performance from evaluation.

Treat recording and review as different modes

During recording, the interface should protect momentum. The user needs a clear prompt, a visible recording state, and one obvious way to stop. Metrics, waveforms, scores, and comparison controls can wait.

During review, the interface can become analytical. The user is no longer trying to stay in character, so it is safe to expose playback controls, a reference track, retry options, and a small review checklist.

This separation sounds minor, but it changes the user's mental model:

Perform the scene without self-editing.
Listen after the take is complete.
Choose one adjustment for the next take.

The third step matters. If a review screen produces ten suggestions, the next performance becomes overloaded again. One specific adjustment—slower pacing, a stronger final word, or a longer pause—is usually more actionable.

Model the experience as a state machine

Browser media tools become easier to reason about when the product has explicit states instead of a collection of loosely related booleans.

A minimal session might use:

idle -> requesting_permission -> ready -> recording -> processing -> review
                                      ^                         |
                                      |-------- retry ----------|

Each state should have a single primary action. This prevents common interface contradictions such as showing both "Record" and "Play" while the browser is still finalizing a media blob.

It also makes error recovery clearer:

Permission denied returns to an explanatory state, not a dead recorder.
A missing microphone produces a device-specific message.
A failed encoding attempt keeps the original take whenever possible.
A retry starts a new take without destroying the previous one until the new take is valid.

The implementation can use a reducer, a finite-state library, or a carefully typed union. The tool is less important than making invalid combinations impossible.

Keep feedback concrete and audible

Voice practice benefits from feedback that the user can hear rather than abstract scores alone. A numeric "expressiveness" value may feel precise, but it does not tell the performer what to change.

Useful review controls include:

Replay the user's take without visual distraction.
Jump directly to the beginning of a line.
Compare two takes at matched volume.
Play a reference performance separately.
Add a short note describing the next intentional change.

A reference should be treated as an interpretation, not the answer. The goal is to help users notice pacing, emphasis, energy, and consistency while preserving their own performance choices.

Design for short, repeatable loops

Long recordings raise the cost of every mistake. Short scenes and clearly bounded lines make practice easier to repeat, especially for new users.

A productive loop can be less than a minute:

Read the character cue.
Record one short take.
Replay it once.
Select one change.
Record the next take.

This loop also reduces technical risk. Shorter media blobs are faster to finalize, easier to retain locally, and less frustrating to retry after a device interruption.

When experimenting with this pattern, I use ChoicerVoicer as a practical browser-based example: the experience centers on performing voice challenges and reviewing the result, instead of presenting recording as an isolated utility.

Make microphone permission part of the product

Permission prompts are often treated as browser plumbing, but they are a critical moment in the user journey. Asking for microphone access immediately on page load removes context and can reduce trust.

A clearer sequence is:

Explain why the microphone is needed.
Let the user initiate the action.
Request access in direct response to that action.
Confirm the selected input device.
Provide a path to recover if permission was denied.

The UI should never imply that audio is being recorded before the browser has actually granted access. Likewise, a glowing microphone icon should reflect a real recording state rather than an optimistic click state.

Preserve the user's sense of control

Voice recordings feel personal. Even when a session is lightweight, the interface should say what is happening to the audio and give users a clear way to discard a take.

Good defaults include keeping an unfinished take local, labeling uploads explicitly, and avoiding surprise sharing. If a recording will leave the device, that transition should be visible and intentional.

This is also why the review screen should not automatically publish, score, or compare a take before the user knows the recording has ended successfully.

The product lesson

The strongest voice-practice experience is not the one with the most analysis on screen. It is the one that helps users alternate cleanly between two mindsets:

Performance mode: commit to the line.
Evaluation mode: observe the result.
Iteration mode: change one thing and try again.

Once these modes are explicit, both the interface and the code become easier to design. The recorder has fewer ambiguous states, feedback becomes more actionable, and the user can focus on performing instead of operating software.

Custom Audio Control on Linux: Switch Between Headphones, Speakers, and Dual Output with a Single Co

Sat, 20 Dec 2025 21:30:50 +0000

Custom Audio Control on Linux: Switch Between Headphones, Speakers, and Dual Output with a Single Command

Introduction

Linux users know that when it comes to system-level control, almost everything is possible — including building your own solutions for problems that graphical interfaces don’t solve well.

In this article, I introduce audioctl, a small Bash script that allows you to switch, with a single command, between:

Headphones connected via HDMI
Analog speakers
Both outputs at the same time (dual mode)

All of this is done using pactl, a bit of Bash, and the flexibility of the Linux audio stack.

TL;DR:
With less than 60 lines of Bash, I built a custom audio controller that gives me more flexibility than any GUI tool.

The Problem

My setup is simple and very common:

My headphones are connected to the monitor, which connects to the PC via HDMI
My speakers are connected to the analog audio output of the motherboard

My father-in-law needed to switch between headphones and speakers — something already possible via the desktop UI — but he also needed to listen to audio on both devices at the same time, for accessibility reasons.

That’s when I thought:

“If Linux gives me full system control, why not build my own audio controller?”

What Is `pactl` and Why It Matters

pactl (PulseAudio Control) is a command-line utility used to control audio servers such as:

PulseAudio
PipeWire (fully compatible and now the default on many distributions)

With pactl, you can:

List audio devices (sinks and sources)
Change volume levels
Switch default outputs
Move audio streams between devices
Load audio modules like module-combine-sink

It’s essentially the Swiss Army knife of Linux audio — and the foundation of this entire solution.

Prerequisites

Before continuing, make sure you have:

A modern Linux distribution (Ubuntu, Fedora, Arch, NixOS, etc.)
PulseAudio or PipeWire
A terminal and text editor
At least two working audio outputs

Tip: To check which audio server you’re using:
pactl info | grep "Server Name"

Step 1: Understand Your Audio Setup

Before automating anything, you need to understand which audio devices are available.

List all sinks (audio outputs):

pactl list sinks short

Example output:

52 alsa_output.pci-0000_01_00.1.hdmi-stereo     RUNNING
57 alsa_output.pci-0000_09_00.6.analog-stereo   SUSPENDED

RUNNING → currently in use
SUSPENDED → available but idle

List audio cards:

pactl list cards

Here you can verify active profiles such as output:analog-stereo.

Step 2: Basic Volume Control with `pactl`

pactl provides a very useful alias:

@DEFAULT_SINK@

# Increase volume
pactl set-sink-volume @DEFAULT_SINK@ +5%

# Decrease volume
pactl set-sink-volume @DEFAULT_SINK@ -5%

This allows you to build volume controls that work regardless of the active device.

Step 3: Switching Audio Outputs

Set a default sink:

pactl set-default-sink SINK_NAME

Move active audio streams manually:

pactl list short sink-inputs
pactl move-sink-input STREAM_ID NEW_SINK

These commands are core building blocks for the final script.

Step 4: Playing Audio on Two Devices at the Same Time (Dual Mode)

This is where things get interesting.

pactl load-module module-combine-sink   sink_name=combined_output   slaves=alsa_output.hdmi,alsa_output.analog   sink_properties=device.description="Dual_HDMI+Analog"

This creates a virtual audio sink that mirrors sound to multiple outputs simultaneously.

Note: PipeWire maintains full compatibility with this module.

Step 5: The Final Script — `audioctl`

The final script wraps all of these commands into simple actions like:

audioctl hdmi
audioctl analog
audioctl dual
audioctl toggle

Full source code:
https://gist.github.com/igorgbr/8031267dee77acf6d7d298ec72a37f94

Installing the Script

chmod +x ~/projects/audioctl
mkdir -p ~/bin
ln -sf ~/projects/audioctl ~/bin/audioctl

Ensure ~/bin is in your PATH:

export PATH="$HOME/bin:$PATH"

Reload your shell:

source ~/.bashrc

Extra Tips

Test with continuous audio:

speaker-test -t wav -c 2

Keyboard shortcuts (GNOME, KDE, i3, etc.):

Super + F11 → audioctl toggle
Super + F12 → audioctl dual

Debug Commands & References

Useful debug commands:

pactl list sinks short
pactl get-default-sink
pactl list short sink-inputs
pactl list cards

Official documentation:

PulseAudio CLI:
https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/CLI/
PipeWire:
https://pipewire.org/
module-combine-sink:
https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Modules/#index7h2

Conclusion

With less than 60 lines of Bash, I solved a real-world problem and gained a level of audio control that no graphical application offered me.

More importantly, this project helped me deeply understand how Linux audio works under the hood and how small automations can dramatically improve everyday workflows.

✨ The control is yours — and it lives in the terminal.

FFmpeg Beginner Guide

Wed, 22 Oct 2025 05:12:36 +0000

Introduction

FFmpeg is one of the most powerful open-source tools for handling multimedia files. Whether you need to convert, compress, record, or stream audio and video, FFmpeg is the go-to command-line utility trusted by developers, content creators, and even large companies.

It supports almost every audio and video format, making it an essential tool in video editing, streaming, and automation pipelines.

Brief History

Released: December 20, 2000
Creator: Fabrice Bellard
Maintained by: FFmpeg team & many open-source contributors

The name FFmpeg comes from:

FF → "Fast Forward"
MPEG → a popular video compression standard (Moving Picture Experts Group)

Since its release, FFmpeg has become a backbone for many popular software and platforms including VLC Media Player, YouTube, and OBS Studio.

Why Use FFmpeg?

Convert videos between formats (MP4, MKV, AVI, MOV, etc.)
Extract or convert audio (MP3, WAV, AAC, etc.)
Compress media without losing much quality
Capture screenshots from videos
Stream audio/video over networks
Automate video/audio processing in scripts

Installation

On Linux (Fedora/Ubuntu):

sudo dnf install ffmpeg   # Fedora
sudo apt install ffmpeg   # Ubuntu/Debian

On macOS (with Homebrew):

brew install ffmpeg

On Windows:

Download builds from ffmpeg.org
Add ffmpeg/bin to your PATH

Basic Commands to get started

1. Video-based Commands

Convert a video to another format

ffmpeg -i input.mp4 output.mkv

It converts an MP4 file into MKV format.

Extract audio from a video

ffmpeg -i video.mp4 -q:a 0 -map a audio.mp3

Takes the audio from a video and saves it as MP3.

Compress a video

ffmpeg -i input.mp4 -vcodec libx265 -crf 28 output.mp4

Uses the H.265 codec with CRF (quality factor) for compression.

Capture a frame (screenshot) from a video

ffmpeg -i video.mp4 -ss 00:00:05 -vframes 1 screenshot.png

Takes a screenshot at the 5-second mark.

Record screen (Linux example)

ffmpeg -video_size 1920x1080 -framerate 30 -f x11grab -i :0.0 output.mp4

Records the desktop screen.

2. Image-based Commands

Convert PNG to JPG

ffmpeg -i input.png output.jpg

In the same way we can convert to other types too.

Convert multiple images into a video

ffmpeg -framerate 24 -i img%03d.png output.mp4

If you have files like img001.png, img002.png..., this creates a video at 24 fps.

Extract all frames from a video as images

ffmpeg -i video.mp4 frame_%04d.png

It saves every frame as an image: frame_0001.png, frame_0002.png...

Resize an image

ffmpeg -i input.png -vf scale=800:600 resized.png

It resizes an image to 800×600 pixels.

Convert image format batch (all PNG → JPG)

for f in *.png; do ffmpeg -i "$f" "${f%.png}.jpg"; done

Converts all .png files in a folder into .jpg. (Linux/Mac shell loop)

Conclusion

FFmpeg may seem intimidating at first because it’s command-line based, but once you learn a few commands, you can convert, compress, and manipulate media effortlessly.

It’s free, fast, and reliable, so no wonder it powers some of the biggest platforms in the world.

SCR4K: The Flagship of Modern Screen Recordings

Fri, 17 Oct 2025 07:30:35 +0000

In a world where precision and performance define professional content creation, Screen Capture Recorder 4K Chrome Extension (SCR4K) stands as the benchmark for clarity and reliability. Built with a lightweight structure, SCR4K, part of the Professional Screen Capture Suite, delivers uncompromising 4K quality without the heavy processing footprint typical of recording tools. This efficiency makes it ideal not only for creators but also for developers seeking a recording engine that integrates seamlessly with modern web environments.

What makes SCR4K exceptional is its universal adaptability. It functions smoothly across Windows, macOS, and Linux, ensuring consistent performance regardless of platform. Its design philosophy prioritizes minimal system impact while maintaining maximum output fidelity—an achievement that allows even lower-powered systems to capture and encode 4K visuals without lag. For professionals who demand smooth, real-time recording and crisp screenshots, SCR4K provides a dependable and scalable solution.

For developers, SCR4K is more than an extension—it’s an integration-ready component. With its straightforward API compatibility and browser-based structure, it can be embedded into any recording or streaming web application within minutes. This flexibility opens doors for SaaS platforms, educators, and enterprise developers to integrate high-definition capture tools into their systems effortlessly. SCR4K isn’t just another recorder—it’s the lightweight core of the next generation of visual content tools, ready to elevate any project today!

Coder Legion - Recent posts tagged audio

Building a Low-Latency Virtual Piano and Drum Kit on iOS with AVAudioEngine

1. The three things a virtual instrument must get right

2. Session setup for an instrument

3. Pitched instruments: AVAudioUnitSampler

4. Drums: preloaded buffers and a player-node pool

5. A multi-touch keyboard that feels like a keyboard

6. Teaching note names and positions

7. Real-world example: Drums n Piano

8. Before you ship a virtual instrument: a checklist

Wrapping up

Building a Real-Time Mic-to-Speaker App on iOS with AVAudioEngine

1. The problem: what "mic to speaker" really needs

2. Microphone permission, the modern way

3. Configuring AVAudioSession

Category and options

Mode

Buffer duration and sample rate

Echo-cancelled input (iOS 18.2+)

Temporary speaker override

Putting it together

4. The AVAudioEngine graph

Format matching

The complete class

5. Taming feedback and latency

Headphones vs. speaker

Voice processing

Buffer size trade-offs

6. Interruptions and route changes

Interruptions

Route changes

Media services reset

Putting it together

7. Background audio and App Store notes

Background audio

App Store review considerations

8. Real-world example: Virtual Mic, a.k.a. Mic to Speaker

Android sidebar: Virtual Mic on Google Play

9. Before you ship live audio: a 7-point checklist

Wrapping up

Text Plus Audio, Fully Offline: Building a Prayer App with AVAudioPlayer and Lock-Screen Controls

1. What "offline text plus audio" requires

2. Bundle the content with a cue sheet

3. AVAudioPlayer and a session that keeps playing

4. Highlighting the current verse

5. Lock Screen and Control Center

6. Interruptions and route changes

7. Real-world example: Hanuman Chalisa : Offline

8. Before you ship a text-plus-audio app: a checklist

Wrapping up

A Practical QA Checklist for Browser-Based Song Search

Start with an input checklist

Separate capture problems from recognition problems

Keep a lightweight test log

A candidate still needs a human check

Try the workflow

Three audio checks worth running before you ship any upload feature

1. Read the header, not the extension

2. Sample rate mismatches are silent

3. Peak level is not loudness

Kempo: Real-Time Audio Time-Stretching, Pitch-Shifting, and Formant Control in Pure KMP

Zero-allocation streaming, phase-locked clarity, and multichannel spatial coherence across JVM, Android, iOS, and WebAssembly.

What Kempo Brings to Your Audio Stack

How Kempo Keeps Audio Crisp (Without the Artifacts)

1. Eliminating Transient Smearing with Vertical Phase-Locking

2. Fixing the "Chipmunk" Effect with Formant Decoupling

3. Preserving Stereo Width and 5.1 Surround Imaging

Getting Started

1. The One-Liner: Offline Track Processing

2. Zero-Allocation Real-Time Streaming

3. Sample-Accurate Offline Rendering with exact()

4. Custom Frequency Mapping & Tonality Limits

Engineered for Speed in Pure Kotlin

Try It Live: The Compose Desktop Studio App

Explore the Code

How AI Music Generators Turn a Text Prompt Into Audio

Step One: Turn Audio Into Tokens

Step Two: Generate the Tokens

Step Three: Condition on the Prompt

Running It Yourself

3. Sample-Accurate Offline Rendering with `exact()`

What Is `pactl` and Why It Matters

Step 2: Basic Volume Control with `pactl`

Step 5: The Final Script — `audioctl`