Coder Legion - Recent posts tagged sqlite

Offline-First Bible Reader on iOS: Bundled SQLite, Full-Text Search and iCloud-Synced Notes

Sun, 25 Oct 2026 12:40:54 +0000

Originally published at https://bestlatestapps.com/blog/building-an-offline-first-bible-reader-on-ios/

How I'd structure a scripture app that works on a plane: a read-only text database, fast search, user data that follows you between devices, read-aloud and a verse-of-the-day widget

The first app I ever published under my App Store developer name was a Bible reader. It went live on 20 April 2020, a day ahead of the next few apps I shipped that month, and today it's called Bible Study AI – Read Offline. It has grown a lot since then: reading plans, devotionals, a prayer journal, widgets and an AI study assistant. But the promise in the name has never changed. It works offline.

"Offline" sounds like a feature you tick off. It isn't. It's an architectural decision, and it shapes everything: where the text lives, how search works, where highlights and notes are saved, how sync behaves when the network comes back, and what you do on a device that has never been online since the app was installed.

This article walks through the architecture I'd recommend for this kind of app: a large, read-only text corpus plus a small amount of precious user data. Scripture is the example, but the same shape fits dictionaries, manuals, legal codes, recipe books and course material. All the code is Swift, and I've checked API names and availability against Apple's developer documentation. Where I'm describing the approach I'd use rather than the app's exact internals, I say so.

1. The problem: two kinds of data with opposite needs

A Bible app holds two very different kinds of data.

The text itself. All 66 books, in several translations; the app's listing mentions KJV, ASV, Webster and World English. That's tens of thousands of verses per translation. It's large, it never changes on the device, and it must be searchable in milliseconds. It belongs to you, the developer, not the user.

What the user creates. Highlights in different colours, notes on verses and chapters, bookmarks, reading-plan progress, prayer journal entries. It's small, it changes all the time, and it's the only data the user would be upset to lose. It belongs to them, and they expect it on their iPad as well as their iPhone.

The mistake I see most often is putting both in one store. Then every app update that fixes a typo in the text has to migrate a database full of user notes, and every sync pushes megabytes of scripture to iCloud for no reason.

So split them:

App bundle (read-only)                 User store (read-write, synced)
┌──────────────────────────┐           ┌──────────────────────────────┐
│ bible.sqlite             │           │ Core Data + CloudKit         │
│  translations, books,    │  verse    │  Highlight, Note, Bookmark,  │
│  chapters, verses        │◀─ id ────▶│  PlanProgress, Prayer        │
│  verses_fts (FTS5)       │           │                              │
└──────────────────────────┘           └──────────────────────────────┘

The only thing that links them is a stable verse identifier. Get that right first.

2. A verse ID that will never change

A verse needs an identifier that's identical across translations, app versions and devices. I use a simple integer: book * 1_000_000 + chapter * 1_000 + verse. John 3:16 is book 43, so its ID is 43_003_016.

struct VerseRef: Hashable, Codable, Comparable {
    let book: Int      // 1...66, canonical order
    let chapter: Int
    let verse: Int

    var id: Int { book * 1_000_000 + chapter * 1_000 + verse }

    init(book: Int, chapter: Int, verse: Int) {
        self.book = book; self.chapter = chapter; self.verse = verse
    }

    init(id: Int) {
        book = id / 1_000_000
        chapter = (id / 1_000) % 1_000
        verse = id % 1_000
    }

    static func < (lhs: VerseRef, rhs: VerseRef) -> Bool { lhs.id < rhs.id }
}

Why an integer and not a string like "JHN.3.16"? It sorts correctly, ranges are cheap (BETWEEN 43003001 AND 43003999 is "all of John 3"), and it's a compact key in both SQLite and Core Data. A highlight saved against 43003016 shows up in KJV and in World English alike, because the ID describes the position, not the wording.

One caveat from experience: versification isn't identical in every tradition. A few translations number some verses differently. If you add a translation that disagrees, map it onto your canonical scheme when you build the database, never at runtime.

3. Shipping the text as a bundled, read-only SQLite database

I build the text database offline with a script, add bible.sqlite to the app target and open it read-only straight from the bundle. No copying to Documents, no migrations, no risk of corruption.

The schema is tiny:

CREATE TABLE verses (
  translation TEXT NOT NULL,   -- 'KJV', 'ASV', 'WEB', ...
  id          INTEGER NOT NULL, -- VerseRef.id
  text        TEXT NOT NULL,
  PRIMARY KEY (translation, id)
) WITHOUT ROWID;

-- Full-text index (built by the script, shipped in the file)
CREATE VIRTUAL TABLE verses_fts USING fts5(
  text, translation UNINDEXED, id UNINDEXED,
  tokenize = 'unicode61 remove_diacritics 2'
);

Opening it from Swift with the system SQLite library:

import Foundation
import SQLite3

enum TextStoreError: Error { case openFailed(String), queryFailed(String) }

final class BibleTextStore {
    private var db: OpaquePointer?

    init() throws {
        guard let url = Bundle.main.url(forResource: "bible", withExtension: "sqlite") else {
            throw TextStoreError.openFailed("bible.sqlite missing from bundle")
        }
        // immutable=1 tells SQLite the file can never change, so it skips locking.
        let uri = "file:\(url.path)?immutable=1"
        let flags = SQLITE_OPEN_READONLY | SQLITE_OPEN_URI
        guard sqlite3_open_v2(uri, &db, flags, nil) == SQLITE_OK else {
            throw TextStoreError.openFailed(String(cString: sqlite3_errmsg(db)))
        }
    }

    deinit { sqlite3_close(db) }

    func chapter(_ book: Int, _ chapter: Int, translation: String) throws -> [(VerseRef, String)] {
        let sql = "SELECT id, text FROM verses WHERE translation = ? AND id BETWEEN ? AND ? ORDER BY id"
        var stmt: OpaquePointer?
        guard sqlite3_prepare_v2(db, sql, -1, &stmt, nil) == SQLITE_OK else {
            throw TextStoreError.queryFailed(String(cString: sqlite3_errmsg(db)))
        }
        defer { sqlite3_finalize(stmt) }

        let start = VerseRef(book: book, chapter: chapter, verse: 0).id
        sqlite3_bind_text(stmt, 1, translation, -1, SQLITE_TRANSIENT)
        sqlite3_bind_int64(stmt, 2, Int64(start))
        sqlite3_bind_int64(stmt, 3, Int64(start + 999))

        var rows: [(VerseRef, String)] = []
        while sqlite3_step(stmt) == SQLITE_ROW {
            let id = Int(sqlite3_column_int64(stmt, 0))
            let text = String(cString: sqlite3_column_text(stmt, 1))
            rows.append((VerseRef(id: id), text))
        }
        return rows
    }
}

// The C macro SQLITE_TRANSIENT doesn't import into Swift; this is the usual stand-in.
private let SQLITE_TRANSIENT = unsafeBitCast(-1, to: sqlite3_destructor_type.self)

Why raw SQLite and not Core Data for the text? Core Data is great for object graphs you edit. A read-only corpus with a full-text index is a different job, and SQLite's FTS5 does that job directly. You can wrap this in GRDB or another library if you prefer; the shape doesn't change.

4. Search that feels instant

The listing promises "Search across all books, chapters, and verses" and "Advanced search with history". With an FTS5 index shipped in the file, the query is one statement:

extension BibleTextStore {
    struct Hit { let ref: VerseRef; let snippet: String }

    func search(_ query: String, translation: String, limit: Int = 100) throws -> [Hit] {
        // Turn "love one another" into "love* one* another*": prefix match on every word.
        let terms = query
            .split(whereSeparator: { !$0.isLetter && !$0.isNumber })
            .map { "\($0)*" }
            .joined(separator: " ")
        guard !terms.isEmpty else { return [] }

        let sql = """
        SELECT id, snippet(verses_fts, 0, '[', ']', '…', 12)
        FROM verses_fts
        WHERE verses_fts MATCH ? AND translation = ?
        ORDER BY rank LIMIT ?
        """
        var stmt: OpaquePointer?
        guard sqlite3_prepare_v2(db, sql, -1, &stmt, nil) == SQLITE_OK else {
            throw TextStoreError.queryFailed(String(cString: sqlite3_errmsg(db)))
        }
        defer { sqlite3_finalize(stmt) }
        sqlite3_bind_text(stmt, 1, terms, -1, SQLITE_TRANSIENT)
        sqlite3_bind_text(stmt, 2, translation, -1, SQLITE_TRANSIENT)
        sqlite3_bind_int(stmt, 3, Int32(limit))

        var hits: [Hit] = []
        while sqlite3_step(stmt) == SQLITE_ROW {
            hits.append(Hit(ref: VerseRef(id: Int(sqlite3_column_int64(stmt, 0))),
                            snippet: String(cString: sqlite3_column_text(stmt, 1))))
        }
        return hits
    }
}

A few things that matter more than the SQL:

Sanitise the query. FTS5 has its own query syntax; a stray quote or hyphen from the user can cause a syntax error. Splitting on non-alphanumerics and rebuilding the string, as above, avoids that.
Debounce in the UI. Search on every keystroke, but cancel the previous task. A .task(id: query) modifier in SwiftUI with a short Task.sleep at the top gives you that for free.
Run it off the main thread. It's fast, but "fast" on an older iPhone in Low Power Mode isn't the same as on your desk.
Search history is user data, so it goes in the user store (next section), not the read-only file.

5. User data: Core Data with CloudKit mirroring

The listing says "iCloud sync across all your devices… Access your highlights, notes, and bookmarks anywhere". The app supports iOS 16.0 and later, which rules out SwiftData (iOS 17+). The natural fit is NSPersistentCloudKitContainer (iOS 13+), which keeps a normal local Core Data store and mirrors it to the user's private CloudKit database in the background.

That's exactly the offline-first behaviour you want: every write goes to the local store first and works with no network; sync catches up later.

import CoreData

final class UserStore {
    static let shared = UserStore()
    let container: NSPersistentCloudKitContainer

    private init() {
        container = NSPersistentCloudKitContainer(name: "UserData") // UserData.xcdatamodeld

        guard let description = container.persistentStoreDescriptions.first else {
            fatalError("No store description")
        }
        description.cloudKitContainerOptions =
            NSPersistentCloudKitContainerOptions(containerIdentifier: "iCloud.com.example.bible")
        // Needed so the app can react to changes that arrive from other devices.
        description.setOption(true as NSNumber, forKey: NSPersistentHistoryTrackingKey)
        description.setOption(true as NSNumber,
                              forKey: NSPersistentStoreRemoteChangeNotificationPostOptionKey)

        container.loadPersistentStores { _, error in
            if let error { print("Store failed to load: \(error)") }
        }
        container.viewContext.automaticallyMergesChangesFromParent = true
        container.viewContext.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy
    }
}

The Highlight entity is deliberately boring: verseID (Integer 64), translation (String, optional), colorName (String), createdAt (Date) and a UUID id. A few CloudKit rules shape that model, and they catch everyone once:

Every attribute must be optional or have a default value. CloudKit can deliver records in any order and partially.
No unique constraints. CloudKit mirroring doesn't support them. If the user highlights the same verse on two offline devices, you'll get two rows; de-duplicate in code (keep the newest per verseID).
Relationships must be optional and need inverses.
Enable the capabilities: iCloud with CloudKit and your container, plus Background Modes → Remote notifications, so pushes from CloudKit can wake the app to sync.
Deploy the schema to production in the CloudKit Console before you ship. A development schema works on your device and silently fails for users. During development, initializeCloudKitSchema(options:) can create the schema from your model for you.

Saving a highlight is then plain Core Data:

extension UserStore {
    func toggleHighlight(_ ref: VerseRef, color: String) throws {
        let context = container.viewContext
        let request = NSFetchRequest<NSManagedObject>(entityName: "Highlight")
        request.predicate = NSPredicate(format: "verseID == %lld", Int64(ref.id))

        let existing = try context.fetch(request)
        if existing.isEmpty {
            let h = NSEntityDescription.insertNewObject(forEntityName: "Highlight", into: context)
            h.setValue(UUID(), forKey: "id")
            h.setValue(Int64(ref.id), forKey: "verseID")
            h.setValue(color, forKey: "colorName")
            h.setValue(Date(), forKey: "createdAt")
        } else {
            existing.forEach(context.delete) // also cleans up offline duplicates
        }
        try context.save()
    }
}

On the reading screen, fetch highlights for the chapter's ID range and merge them with the verse rows from SQLite in a view model. The two stores never need to know about each other.

6. Listen mode with AVSpeechSynthesizer

The listing's "Listen & Read" section mentions text-to-speech and auto-scroll. On iOS that's AVSpeechSynthesizer (iOS 7+), and its delegate tells you which range of text is about to be spoken, which is all you need to highlight and scroll along.

import AVFoundation

@MainActor
final class ReadAloud: NSObject, ObservableObject, AVSpeechSynthesizerDelegate {
    @Published var currentVerse: VerseRef?
    private let synth = AVSpeechSynthesizer()
    private var refs: [ObjectIdentifier: VerseRef] = [:]

    override init() {
        super.init()
        synth.delegate = self
    }

    func speak(chapter verses: [(VerseRef, String)], language: String = "en-US") {
        synth.stopSpeaking(at: .immediate)
        refs = [:]
        for (ref, text) in verses {
            let u = AVSpeechUtterance(string: text)
            u.voice = AVSpeechSynthesisVoice(language: language)
            u.rate = AVSpeechUtteranceDefaultSpeechRate
            u.postUtteranceDelay = 0.15        // a breath between verses
            refs[ObjectIdentifier(u)] = ref
            synth.speak(u)                     // utterances are queued in order
        }
    }

    func pause() { synth.pauseSpeaking(at: .word) }
    func resume() { synth.continueSpeaking() }

    nonisolated func speechSynthesizer(_ s: AVSpeechSynthesizer,
                                       didStart utterance: AVSpeechUtterance) {
        let key = ObjectIdentifier(utterance)
        Task { @MainActor in
            self.currentVerse = self.refs[key]
        }
    }
}

The view observes currentVerse and calls ScrollViewReader.scrollTo(_:anchor:) with the verse ID. For word-level highlighting, implement speechSynthesizer(_:willSpeakRangeOfSpeechString:utterance:) as well.

Two practical notes. If you want speech to continue with the screen locked ("perfect for listening while commuting"), set the audio session category to .playback and add the audio background mode, and only while speech is actually running. And the voices available vary by device and by what the user has downloaded in Settings, so let them pick, and fall back gracefully when AVSpeechSynthesisVoice(language:) returns nil.

7. Verse of the Day on the Home Screen with WidgetKit

The listing has "Home screen widgets: quick access to verse of the day". A widget runs in its own extension process, so it can't reach into the app's memory. Two things make this simple with the architecture above:

The text database is read-only and in the bundle, so the widget extension can ship its own copy (or you can share it through an App Group container). No sync needed.
The verse of the day is a deterministic function of the date. Every device shows the same verse without a server.

import WidgetKit
import SwiftUI

struct VerseEntry: TimelineEntry {
    let date: Date
    let reference: String
    let text: String
}

struct VerseProvider: TimelineProvider {
    func placeholder(in context: Context) -> VerseEntry {
        VerseEntry(date: .now, reference: "John 3:16", text: "For God so loved the world…")
    }

    func getSnapshot(in context: Context, completion: @escaping (VerseEntry) -> Void) {
        completion(entry(for: .now))
    }

    func getTimeline(in context: Context, completion: @escaping (Timeline<VerseEntry>) -> Void) {
        let now = Date()
        let tomorrow = Calendar.current.startOfDay(for: now.addingTimeInterval(86_400))
        completion(Timeline(entries: [entry(for: now)], policy: .after(tomorrow)))
    }

    private func entry(for date: Date) -> VerseEntry {
        // Pick from a curated list by day-of-year: same verse for everyone, every device.
        let day = Calendar.current.ordinality(of: .day, in: .year, for: date) ?? 1
        let pick = DailyVerses.list[(day - 1) % DailyVerses.list.count]
        return VerseEntry(date: date, reference: pick.reference, text: pick.text)
    }
}

struct VerseOfTheDayWidget: Widget {
    var body: some WidgetConfiguration {
        StaticConfiguration(kind: "VerseOfTheDay", provider: VerseProvider()) { entry in
            VStack(alignment: .leading, spacing: 6) {
                Text(entry.text).font(.callout).lineLimit(5)
                Text(entry.reference).font(.caption).foregroundStyle(.secondary)
            }
            .containerBackground(.fill.tertiary, for: .widget) // iOS 17+
        }
        .configurationDisplayName("Verse of the Day")
        .supportedFamilies([.systemSmall, .systemMedium])
    }
}

DailyVerses.list is a small curated array compiled into the extension. With an iOS 16 deployment target, wrap containerBackground in an availability check, since that modifier is iOS 17+. The .after(tomorrow) policy asks WidgetKit to refresh just after midnight. It's a request, not a guarantee, so the entry should still read correctly if WidgetKit refreshes a bit late.

8. Real-world example: Bible Study AI – Read Offline

All of this comes from building and maintaining my own app, Bible Study AI – Read Offline. The patterns above are the approach I'd use and recommend; I'm describing the architecture, not giving a line-by-line tour of the shipping code.

At the time of writing, its public App Store listing shows:

Free, in the Books and Education categories
First released in April 2020: my first app on the store
113 ratings in the US store, averaging about 4.6 stars
Supports iOS 16.0 and later, on iPhone and iPad

Here's how the listing's features map to the sections above:

What the listing says	Where it's covered
"Read all 66 books of the Bible offline" / "Multiple translations: KJV, ASV, Webster, World English, and more"	Bundled read-only SQLite, one table keyed by translation + verse ID (sections 2–3)
"Search across all books, chapters, and verses" / "Advanced search with history"	FTS5 index shipped in the file, sanitised prefix queries (section 4)
"Highlight verses with multiple colors" / "Add personal notes" / "Bookmark favorite passages"	Small Core Data entities keyed by verse ID (section 5)
"iCloud sync across all your devices"	`NSPersistentCloudKitContainer` mirroring to the private database (section 5)
"Text-to-speech support" / "Auto-scroll for hands-free reading"	`AVSpeechSynthesizer` and its delegate driving the scroll position (section 6)
"Home screen widgets: quick access to verse of the day"	WidgetKit `StaticConfiguration`, date-derived entry (section 7)

The listing also covers reading plans, daily devotionals, a prayer journal with reminders and a "Bible Study Assistant" for AI-powered answers. Those all sit on top of the same split: plan progress and prayers are user data in the synced store, reminders are local notifications, and the assistant is the one feature that needs a network. I've left the AI part out of this article on purpose. It's the one piece that isn't offline, and the most important design rule there is simply to make sure the rest of the app never waits on it.

9. Before you ship an offline-first reader: a checklist

Airplane-mode test from a fresh install. Delete the app, turn on Airplane Mode, install from TestFlight… then read, search, highlight, take a note and listen. Everything except network features must work.
Text and user data are separate. The corpus is read-only in the bundle; user data lives in its own store. An app update that changes the text never touches user data.
Stable IDs everywhere. Every highlight, note and bookmark points at a canonical verse ID, never at a row number or a translation-specific string.
The CloudKit model follows the rules. Optional attributes or defaults, no unique constraints, optional relationships with inverses, and the schema is deployed to production.
Duplicates are handled. Two offline devices editing the same verse produce two records; your code merges them predictably.
Search can't crash on user input. Quotes, hyphens, emoji and other scripts are all tested against your FTS query builder.
Background audio is honest. The audio mode is used only while read-aloud is playing, and the session is deactivated when it stops.
The widget works with no app launch. Install, add the widget, never open the app: it should still show a verse.

Wrapping up

Offline-first sounds like a constraint, but for a reading app it makes things simpler. A read-only database in the bundle can't be corrupted and never needs migrating. A small, separate user store syncs cheaply. And a verse ID that never changes ties the two together. Add AVSpeechSynthesizer and WidgetKit and you get most of what people want from a daily reading app, without a single required network call.

If you want to see where this ended up, try the app:

Bible Study AI – Read Offline (iOS): https://apps.apple.com/us/app/bible-study-ai-read-offline/id1508795537
All of my free iOS apps: https://apps.apple.com/us/developer/alok-singh/id1508795536
My portfolio: https://bestlatestapps.com/alok/

If you've built a large offline corpus on iOS and solved sync or search differently, tell me in the comments. I'm always curious how other people draw the line between "bundled" and "user" data.

— Alok Kumar Singh

Originally published at https://bestlatestapps.com/blog/building-an-offline-first-bible-reader-on-ios/

SQLite's Two-Person Team Just Passed Its First AI Stress Test

Tue, 15 Sep 2026 16:47:36 +0000

This spring, open source maintainers across the internet started noticing the same strange thing at the same time: a flood of bug reports that no human had written. Not spam. Not malicious. Just confidently wrong, at a scale nobody had planned for. D. Richard Hipp, the creator of SQLite, watched it happen to his own project and lived to describe it calmly, a few months later, from his home office in Charlotte.

"This spring, late spring into summer, there was just an avalanche of bug reports that were generated by AIs," Hipp said. "And it's not just us. Every open source project I've talked to has experienced the same thing."

That's worth sitting with for a second, because SQLite isn't a side project. It's the database engine sitting inside your phone, your browser, and most of the apps on both. Hipp built it in 2000 as a Navy contractor, frustrated that he kept getting blamed for outages caused by a database administrator's server, not his code. His fix was to get rid of the server entirely. "You don't want to have to install Postgres in order to run Firefox," he said. Twenty-six years later, that decision is why SQLite runs on something like a trillion devices without anyone thinking about it.

So when the AI bug reports hit, they hit hard, and some of them were real. Most, Hipp said, landed on obscure, untested files that had been sitting in the source tree for years, never meant to ship. A few were serious. What surprised him wasn't the volume. It was how fast it stopped. "It's as if the AIs have discovered all the bugs they're able to discover, and we're through that," he said. "I'm pleasantly surprised that this went away so quickly." Compare that to the profile-guided fuzzing wave a decade ago, which took years to work through. This one burned out in a few months. Hipp isn't declaring victory for AI or against it — he compares it to security conferences like Black Hat, where the same tools that help attackers eventually help defenders more. "That's been true of what we've done for the last 100 years," he said.

Here's the part that should get developers' attention: Hipp isn't avoiding AI. He's using it, carefully, in exactly one place. "I run all the new patches through Claude now," he said. "Claude does code review — it doesn't write any code for it." It catches real logic problems and the small stuff, like a comment nobody updated. That's a meaningful admission from someone who has kept SQLite's core team at five committers — two of whom do nearly all the work — while Postgres runs on 40-plus contributors and DuckDB reportedly staffs 24 people in Amsterdam alone. Hipp says that's only possible because of SQLite's 100% branch test coverage, built by hand over roughly a year of 80-hour weeks. The rigor is what lets two people move fast without breaking things. AI review adds a second set of eyes on top of that, not instead of it.

What AI still doesn't get is a seat writing SQLite's core. "I've tried that, and what I've seen is that the AI-generated code is not up to snuff," Hipp said, then immediately qualified it: "I've noticed even just over the past year that it's getting better. My opinion could possibly change." Right now, though, the answer is no. It's all human-written.

That same restraint shows up in how Hipp treats the ecosystem building on top of him. Forks like libSQL and platforms like Turso and Cloudflare D1 add concurrent writes, replication, and edge distribution — features SQLite deliberately doesn't ship. Hipp isn't threatened by it. "I'm fine with someone else filling it," he said. "It's public domain. You can do that. Do what makes you happy." He's also seen the pattern before: "They tend to be venture capital backed, which means we've seen a lot of them come and go. Mostly gone." He built SQLite for local, embedded state in 2000 for reasons that had nothing to do with AI agents. That it turned out to be exactly what 2026's edge and agent workloads need is a convergence he didn't predict — he's on record dismissing large language models entirely back in the 1980s, for lack of data and compute. "Boy, was I wrong," he said, without much drama about it.

For developers deciding when SQLite is the right call, Hipp's rule is simpler than most decision trees: keep the database and the data on the same device. If your data lives somewhere else, or you're past roughly 17 terabytes, or you need many applications hammering writes at once, reach for a client-server database instead. And if you're already using SQLite, don't just dump JSON into it and call it a day — he sees too many developers treating it like a document store when the relational model would serve them better.

Hipp has pledged to keep maintaining SQLite through 2050. He's about halfway there. What he said about why matters more than the date: thinking in decades instead of toward a release, an IPO, or a VC's milestone "changes the way you code." He's also honest about the one thing that could end the project entirely — not a rival database, but AI good enough to rewrite SQLite from scratch, smaller and faster, with fewer bugs. "We're not there yet," he said. "But that might happen. And when that happens, fine. I'm done."

That's not resignation. It's the same stubbornness that built the thing in the first place, applied one more time to whatever comes next.

Alembic in Practice: Structure, Revisions, and the Traps SQLite Sets

Thu, 10 Sep 2026 03:50:15 +0000

What those hex filenames mean, why migrations silently applied to the wrong database for two months, and how to widen a CHECK constraint without quietly breaking it.

Alembic is the migration tool for SQLAlchemy. Most tutorials get you as far as
alembic revision --autogenerate and stop, which leaves you unprepared for the
two things that actually go wrong: SQLite's refusal to alter constraints, and
configuration that points migrations at a different database than your
application uses.

This post covers both, using real migrations from
FinOps Sentinel, an AWS cost
agent with four migrations, one of which took three attempts to get right.

The layout

alembic.ini                     # config; script location, logging
alembic/
├── env.py                      # runs on every alembic command
├── script.py.mako              # template for new revision files
└── versions/
    ├── c447645246ac_initial_schema.py
    ├── 08872b80af2e_notifications_and_remediations.py
    ├── a1c9f4d27b13_add_rds_and_s3_resource_types.py
    └── b7e2f81c4a90_add_spend_snapshots.py

Plus one table Alembic creates in your database:

CREATE TABLE alembic_version (version_num VARCHAR(32) NOT NULL);

One row. One value. That is the entirety of Alembic's state: which database is
at which revision. Everything else is derived.

Why the filenames look like keyboard mashing

This is the most common first question, and the answer explains the design.

c447645246ac_initial_schema.py
b7e2f81c4a90_add_spend_snapshots.py

The hex prefix is the revision ID: a 12-character slice of a UUID, generated
by alembic revision. The filename is <revision_id>_<your message slugified>.py.

Inside each file:

revision: str = "b7e2f81c4a90"
down_revision: Union[str, None] = "a1c9f4d27b13"

Those two fields form a linked list:

c447645246ac  initial_schema                  (down_revision = None)  ← root
      ↓
08872b80af2e  notifications_and_remediations
      ↓
a1c9f4d27b13  add_rds_and_s3_resource_types
      ↓
b7e2f81c4a90  add_spend_snapshots                                     ← head

Order comes from the pointers, never from filenames. Look at the project's
list again: b7e2f81c4a90 sorts alphabetically before c447645246ac, yet
c447645246ac is the first migration and b7e2f81c4a90 is the newest. If
Alembic sorted by filename, everything would run backwards.

Why not 0001, 0002, 0003?

Because sequential numbers collide the moment two people work in parallel.

main
 ├── feature/billing   → 0004_add_invoices.py
 └── feature/auth      → 0004_add_sessions.py

Both are valid. Both are 0004. Merging produces a conflict where the numbering
lies about order, and worse, the conflict is in the filename, and git resolves it
by keeping both, and now your migration history has two fourth steps.

With hex IDs there is no collision. And if both branches set down_revision to
the same parent, Alembic detects a genuine branch:

$ alembic upgrade head
ERROR: Multiple head revisions are present

It refuses to guess, and you resolve it explicitly with alembic merge. The
scary-looking IDs are what make that detection possible.

If you want readable filenames

Add to alembic.ini:

file_template = %%(year)d%%(month).2d%%(day).2d_%%(rev)s_%%(slug)s

Giving 20260728_b7e2f81c4a90_add_spend_snapshots.py: date-sortable, revision
ID still present because Alembic needs it. Existing files keep their names; only
new ones pick it up.

The configuration trap that cost two months

This is the bug I most want to warn people about, because it produces no
error message at all.

The default env.py reads the database URL from alembic.ini. FinOps Sentinel
needed it configurable, so early on it did the obvious thing:

# alembic/env.py: the version with the bug
def get_url():
    db_path = os.getenv("SENTINEL_DB_PATH", ".sentinel.db")
    return f"sqlite:///{db_path}"

Reasonable. Except the application reads the same setting through
pydantic-settings:

class Settings(BaseSettings):
    sentinel_db_path: str = ".sentinel.db"
    model_config = SettingsConfigDict(env_file=".env")   # ← reads .env too

os.getenv does not read .env. pydantic-settings does.

The documented setup puts SENTINEL_DB_PATH=data/sentinel.db in .env, not in
the shell. So:

Component	Reads from	Resolves to
The app	env var → `.env` → default	`data/sentinel.db`
Alembic	env var → default	`.sentinel.db`

Every migration ever run landed on .sentinel.db. The application used
data/sentinel.db. Two files, silently, for two months.

Why nothing broke for so long

The application also calls Base.metadata.create_all() in tests, and the first
three migrations only created tables that SQLAlchemy would have created anyway.
The schemas happened to agree.

Then came a migration whose table nothing else creates: spend_snapshots. The
next scan died:

alembic upgrade head
INFO  [alembic.runtime.migration] Running upgrade a1c9f4d27b13 -> b7e2f81c4a90

sentinel scan
OperationalError: (sqlite3.OperationalError) no such table: spend_snapshots

Alembic reported success. The app said the table didn't exist. Both were telling
the truth about different files.

The fix: one owner for the URL

# config.py: the single source of truth
def database_url() -> str:
    """The one place the findings database URL is built.

    Both the application (bootstrap.get_repository) and Alembic (alembic/env.py)
    call this. They used to build it independently, and drifted.
    """
    return f"sqlite:///{settings.sentinel_db_path}"

# alembic/env.py
from finops_sentinel.config import database_url

def get_url():
    return database_url()

# bootstrap.py
def get_repository() -> FindingsRepository:
    return SqlAlchemyRepository(db_url=database_url())

And alembic.ini is deliberately emptied:

# Deliberately empty. env.py overrides this with the application's own
# Settings.sentinel_db_path. A URL here would be dead config that reads as
# authoritative, which is exactly how this drifted before.
sqlalchemy.url =

Takeaway: if your app and your migrations resolve the database path
separately, they will eventually disagree, and nothing will tell you. Give the
URL exactly one owner and have both call it.

The project now pins this with tests:

def test_the_app_and_alembic_build_the_url_the_same_way():
    assert get_repository().db_url == database_url()

def test_alembic_ini_does_not_hardcode_a_database():
    parser = configparser.ConfigParser()
    parser.read(REPO_ROOT / "alembic.ini")
    assert parser.get("alembic", "sqlalchemy.url", fallback="").strip() == ""

def test_alembic_env_does_not_read_the_environment_itself():
    env_py = (REPO_ROOT / "alembic" / "env.py").read_text()
    assert "database_url" in env_py
    assert "os.getenv(" not in env_py

Autogenerate, and where to stop trusting it

alembic revision --autogenerate -m "add spend snapshots"

Alembic compares your SQLAlchemy models against the live database and writes the
diff. It needs target_metadata wired up in env.py:

from finops_sentinel.adapters.persistence.sqlalchemy_repo import Base
target_metadata = Base.metadata

It detects new/dropped tables and columns, nullability changes, most index
and unique-constraint changes, and some type changes.

It misses table and column renames (it sees a drop plus an add, and will
happily destroy your data), CHECK constraint changes, server defaults in many
cases, and anything requiring data transformation.

The generated file is a draft. The comment Alembic itself writes says so:

def upgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###

Always read it. For anything beyond adding a column, expect to hand-write.

Trap: SQLite cannot ALTER a constraint

Adding two values to an enum required widening a CHECK constraint:

class ResourceType(StrEnum):
    EBS_VOLUME = "ebs_volume"
    ELASTIC_IP = "elastic_ip"
    EC2_INSTANCE = "ec2_instance"
    EBS_SNAPSHOT = "ebs_snapshot"
    RDS_INSTANCE = "rds_instance"    # new
    S3_BUCKET = "s3_bucket"          # new

On Postgres: ALTER TABLE ... DROP CONSTRAINT ... ADD CONSTRAINT .... Done.

On SQLite: there is no ALTER for constraints. The only route is create a new
table, copy the data, drop the old, rename the new. Alembic wraps this in
batch_alter_table:

with op.batch_alter_table("resources", recreate="always") as batch_op:
    ...

Attempt 1: silently kept the old constraint

The first version passed table_args and ran without error:

with op.batch_alter_table(
    "resources",
    recreate="always",
    table_args=(
        sa.CheckConstraint(_resource_type_check(NEW_TYPES), name="check_resource_type"),
    ),
):
    pass

alembic upgrade head printed success. Inserting an rds_instance row still
failed.

Why: table_args adds to a reflected table definition rather than
replacing it. Alembic reflected the live table, including the old four-value
CHECK, then appended the new six-value one. The rebuilt table had both, and
the old one still rejected every new row.

Only dumping the schema revealed it:

sqlite3 data/sentinel.db ".schema resources"
# CONSTRAINT check_resource_type CHECK (resource_type IN ('ebs_volume', ...))   ← old
# CONSTRAINT check_resource_type CHECK (resource_type IN ('ebs_volume', ..., 's3_bucket'))

Attempt 2: `copy_from` to suppress reflection

The fix is copy_from, an explicit table definition that tells Alembic not
to reflect:

def _columns_only_resources() -> sa.Table:
    """The `resources` table as columns and primary key, with NO constraints.

    Handed to batch_alter_table as `copy_from`, which suppresses reflection of
    the live table. That suppression is the point: `table_args` adds to a
    reflected definition rather than replacing it.
    """
    return sa.Table(
        "resources",
        sa.MetaData(),
        sa.Column("id", sa.String(), primary_key=True, nullable=False),
        sa.Column("resource_id", sa.String(), nullable=False),
        sa.Column("resource_type", sa.String(), nullable=False),
        sa.Column("resource_arn", sa.String(), nullable=False),
        sa.Column("region", sa.String(), nullable=False),
        sa.Column("current_tags", sa.String(), nullable=False),
        sa.Column("lifecycle", sa.String(), nullable=False),
        sa.Column("first_seen_at", sa.DateTime(timezone=True), nullable=False),
        sa.Column("last_seen_at", sa.DateTime(timezone=True), nullable=False),
    )

Attempt 3: the indexes were gone

copy_from suppresses index reflection too. The rebuilt table lost its
unique index on resource_id, which the upsert logic depends on for
correctness. So it gets recreated by hand:

def _rebuild_resources(resource_types: tuple[str, ...]) -> None:
    with op.batch_alter_table(
        "resources",
        recreate="always",
        copy_from=_columns_only_resources(),
        table_args=(
            sa.CheckConstraint("lifecycle IN ('active', 'deleted')",
                               name="check_resource_lifecycle"),
            sa.CheckConstraint(_resource_type_check(resource_types),
                               name="check_resource_type"),
        ),
    ):
        pass  # the rebuild itself is the migration

    # copy_from also suppresses index reflection, so re-create by hand.
    bind = op.get_bind()
    existing = {index["name"] for index in sa.inspect(bind).get_indexes("resources")}
    if "ix_resources_resource_id" not in existing:
        op.create_index(op.f("ix_resources_resource_id"), "resources",
                        ["resource_id"], unique=True)

Rule for SQLite batch migrations: whatever the batch block omits is silently
absent from the rebuilt table. Every column, every constraint, every index has
to be re-declared. And "it ran without error" proves nothing. Dump the schema.

Downgrades that refuse to run

Most downgrade() functions are written on autopilot as the inverse of
upgrade(). Sometimes the honest inverse is refusing:

def downgrade() -> None:
    bind = op.get_bind()
    stranded = bind.execute(
        sa.text("SELECT COUNT(*) FROM resources "
                "WHERE resource_type IN ('rds_instance', 's3_bucket')")
    ).scalar_one()
    if stranded:
        raise RuntimeError(
            f"Cannot downgrade: {stranded} resource(s) use rds_instance or s3_bucket, "
            "which the previous CHECK constraint forbids. Delete those resources and "
            "the findings referencing them first if this downgrade is intended."
        )
    _rebuild_resources(OLD_RESOURCE_TYPES)

Narrowing the constraint with those rows present means deleting them, which
also orphans every finding referencing them. A downgrade that silently destroys
data to succeed is worse than one that stops and explains.

The docstring says so up front, so nobody is surprised at 2am:

downgrade() restores the four-value constraint and will FAIL if any
rds_instance or s3_bucket row exists by then, deliberately.

A simple migration, for contrast

Not every migration is a fight. Adding a table is exactly what you'd hope:

def upgrade() -> None:
    op.create_table(
        "spend_snapshots",
        sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True, nullable=False),
        sa.Column("snapshot_date", sa.Date(), nullable=False),
        # Stored as a string, matching SafeNumeric: SQLite has no Decimal, and
        # a float column would quietly round money.
        sa.Column("total_estimated_monthly_usd", sa.String(), nullable=False),
        sa.Column("open_findings", sa.Integer(), nullable=False),
        sa.Column("active_resources", sa.Integer(), nullable=False),
        sa.Column("captured_at", sa.DateTime(timezone=True), nullable=False),
    )
    op.create_index(
        op.f("ix_spend_snapshots_snapshot_date"),
        "spend_snapshots", ["snapshot_date"], unique=True,
    )

Note the docstring convention this project uses: the why, not the what:

snapshot_date is UNIQUE on purpose. The repository upserts by date, so two
scans on the same day collapse to one row. Otherwise a day that happened to
be scanned five times would weigh five times as much in the trailing mean, and
the z-score would measure scan cadence rather than spend.

Six months later, that paragraph is worth more than the code.

Keeping constraints from drifting

CHECK constraints duplicating enum values is a drift hazard. This project
generates them:

def _in_clause(column: str, values: type[StrEnum]) -> str:
    """Render `column IN ('a', 'b', ...)` straight from a StrEnum.

    Generated rather than hand-written so the constraint cannot drift from the
    enum: adding a ResourceType now updates this automatically, and only the
    Alembic migration has to be written by hand.
    """
    members = ", ".join(f"'{member.value}'" for member in values)
    return f"{column} IN ({members})"


class ResourceModel(Base):
    __tablename__ = "resources"
    ...
    __table_args__ = (
        CheckConstraint(_in_clause("lifecycle", ResourceLifecycle),
                        name="check_resource_lifecycle"),
        CheckConstraint(_in_clause("resource_type", ResourceType),
                        name="check_resource_type"),
    )

The model side can never drift from the enum. Only the migration is
hand-written, which is unavoidable, since a migration must describe a transition between
two states, and only one of them exists in your code.

Testing migrations

A green alembic upgrade head proves almost nothing (see attempt 1). Two things
worth doing:

1. Round-trip against a copy of a real database.

cp data/sentinel.db /tmp/mig-test.db
SENTINEL_DB_PATH=/tmp/mig-test.db alembic upgrade head
SENTINEL_DB_PATH=/tmp/mig-test.db alembic downgrade -1
SENTINEL_DB_PATH=/tmp/mig-test.db alembic upgrade head

sqlite3 /tmp/mig-test.db "select count(*) from resources; select count(*) from findings;"
# 274 / 187, same as before

Real data, real row counts, and it catches data loss a fresh database never
would.

2. Assert the error message is useful.

An un-migrated database should not produce a raw SQL traceback:

def upsert_resource(self, resource: Resource) -> None:
    try:
        self._upsert_resource(resource)
    except IntegrityError as exc:
        if "check_resource_type" not in str(exc):
            raise
        raise RuntimeError(
            f"The findings database does not allow resource type "
            f"'{resource.resource_type}'. It was created by an older "
            f"version of FinOps Sentinel. Run `alembic upgrade head` "
            f"against {self.db_url} and scan again."
        ) from exc

Failing hard is right: a half-written inventory is worse than a stopped run.
But the traceback told nobody what to do. Now it names the type, the database,
and the command.

Command reference

alembic revision -m "message"                  # empty migration, hand-written
alembic revision --autogenerate -m "message"   # diff models vs database

alembic upgrade head          # apply everything outstanding
alembic upgrade +1            # one step forward
alembic downgrade -1          # one step back
alembic downgrade base        # all the way down

alembic current               # what revision is this database at?
alembic history --verbose     # the full chain
alembic heads                 # >1 means you have a branch to merge
alembic merge -m "merge" a1b2 c3d4

alembic upgrade head --sql    # print SQL instead of executing (offline mode)

alembic current is the one to reach for when something is confusing. It tells
you where Alembic thinks the database is, and if that disagrees with the
database you expect, you have the bug from earlier in this post.

Checklist

Never edit a migration that has been applied anywhere but your laptop.
Write a new one. The old revision ID is a foreign key in every database that
ran it.
Read autogenerated migrations before committing. Especially for renames,
which autogenerate turns into drop-plus-add.
Write the "why" in the docstring. The SQL says what; only you can say why.
Test on a copy of real data. Fresh databases hide data-loss bugs.
Dump the schema after non-trivial migrations. "It ran" is not "it worked."
Give the database URL one owner that both the app and Alembic call.
Make downgrades honest, including refusing when the honest inverse is
data loss.

Migrations are the one part of a codebase where a mistake is not just wrong but
permanently wrong, replicated to every environment that ran it. Worth the
extra ten minutes.

Every example here is from github.com/boazleleina/finops-sentinel

Shrinking a Materials Science Dataset

Tue, 08 Sep 2026 11:37:32 +0000

Shrinking datasets is quite contextual. It depends on how the dataset is going to be used. Its the same concept as optimizing databases, based on the read/write patterns.

PACLEC and Traffic Patterns are the key determining factors.

Hosted on fly.io : https://materials-db.fly.dev/

The materials trajectory database was around 12Gigs and is mostly used for training models like CHGNet.
But I needed it only for responding to query on finding materials and it didn't need all the variations.

The aim was to be able to serve it using my most loved database - sqlite, in under reasonable timeframe: <500ms.

Summary:

The source data and the problem. The project is a Go API serving 145,923 materials from the Materials Project, backed by a single SQLite file. The source data is 11.35GB, while the database it actually runs on is only 266MB. The raw dataset (MPtrj) is so large because it was built as a training corpus for a machine-learning model - each material's relaxation run gets sampled at several points along a trajectory at different pressures and energies, making every point a valid training example. That's why the dataset has 1.58 million rows, not because of sloppy deduplication.

KeySpace Reduction: Most of the times JSON has long key names, this is unnecessary to storage, so converting keys like "lattice_structure" to "ls" was the first step. This was the initial memory optimization that was needed. The versioned response adapter would expand these back

The big win: deduplication (~20x reduction). The first and largest cut came purely from scope - the ingest step keeps only one structure per material, picked by lowest energy, using a ROW_NUMBER() window function. Every other task retains only five scalar properties in a separate table, but loses its full structure blob. This single decision takes the dataset from 11.35GB down to roughly 592MB. This isn't "free cleanup" - those discarded structures had real scientific value for ML training, but the API's job (ground-state lookups) simply didn't need them.

Compression: 592MB -> 266MB. The remaining bulk was almost entirely structure_json blobs - one per material. Two techniques were applied: first, the verbose pymatgen JSON format was stripped down to a compact positional array (lattice matrix + sites as [element, a, b, c, occupancy]), and then zlib compression was applied on top. Benchmarking showed that stripping first and then compressing outperformed either approach alone. Each blob also got a 4-byte versioned header so old and new rows could coexist during migration without requiring a full re-ingest.

Serving it fast. At 266MB, the entire database fits inside a single 1GB mmap region, so reads hit the page cache after the first touch. A channel-based pool of read-only connections runs warmup queries against key indexes on boot, singleflight collapses concurrent identical requests into one SQLite call, and an LRU cache sits in front of lookups since the data is static. The honest caveat: a cold start on a small VM still takes 15–24 seconds for the first request, but everything after that is sub-500ms. Browser cache headers do the rest of the job

Apart from singleflight, nginx could also be used, but this is fast enough.

A detailed write up is provided in my blog: https://ikouchiha47.github.io/2026/08/05/materialsdb-11gb-to-266mb.html

Ranking New Papers By Meaning Instead Of Keywords, In One PHP App

Sun, 23 Aug 2026 22:05:46 +0000

Most of us have built some version of a scheduled job that watches a feed and mails out a summary, and it almost never fails on the plumbing. It fails on relevance, and this is a writeup of a small PHP and SQLite project that puts a model scoring step where the keyword filter usually goes.

Why Keyword Alerts Keep Missing The Right Paper

Every major index offers alerts, and they all match on words. That is the whole problem. A paper that applies your exact method in a neighboring field will share none of your keywords, so it never reaches you. A paper that happens to use your words while solving an unrelated problem arrives every single week.

Word overlap is a proxy for relevance, and it is a bad one the moment your work sits between two subfields. Any watcher you build on top of a keyword query inherits that ceiling no matter how good the rest of the code is.

The Shape Of The Pipeline

Literature Radar splits the job into four steps, and the split is the interesting part.

First, an interest profile. Not a keyword list, a short paragraph written the way you would describe your work to a colleague. Separately you define topic queries that tell the poller what to pull, so retrieval stays broad while ranking stays narrow.

Second, a scheduled poll against arXiv, PubMed, Crossref, bioRxiv and medRxiv, writing into one local SQLite file. Anything already seen is skipped, so the table doubles as the dedupe layer.

Third, the ranking pass. Your model reads each new title and abstract and scores it 0 to 10 against the profile, attaching a one line reason. This is where a paper about your method in another field can outscore a paper that matched your keywords exactly.

Fourth, the digest. One email on your schedule, best first, score and reason attached, nothing repeated across sends.

Choosing Where The Ranking Runs

The ranking backend is resolved in order: the Claude command line tool, the Codex command line tool, an Anthropic API key, then any OpenAI compatible endpoint, which covers OpenAI, OpenRouter, and local servers like Ollama or LM Studio.

That order matters for cost. If you already pay for a Claude or ChatGPT subscription, the CLI path means the ranking step adds no per token cost at all, which is what makes a daily job over a few hundred abstracts practical for one person. Falling back to a local model through an OpenAI compatible endpoint keeps abstracts on your own machine entirely.

The whole thing is one PHP app plus one SQLite file, driven by a single cron entry that decides internally whether a poll, a rank, or a digest is due:

0 */2 * * * php /path/to/Literature-Radar/cli/cron.php

Each step also runs standalone, which is what you want while tuning the profile.

What Transfers To Your Own Watcher

Three things generalize past papers. Keep retrieval broad and put the judgment in a separate ranking pass, because a narrow query loses items you can never recover later. Store the reason next to the score, since a ranked list with no explanation gives you nothing to tune against. And make the scoring backend swappable from day one, because the model you start on will not be the one you finish on.

It is MIT licensed with no account and no service in the middle, so the abstracts go to whichever model you picked and nowhere else.

One PHP App and One SQLite File: A Self Hosted Workflow Engine

Fri, 14 Aug 2026 16:23:47 +0000

Most of us here have written the same glue script more than once, the one that catches a form submission and pushes it somewhere else, then quietly rots in a cron job nobody documented. This is a look at an open source engine that gives that glue a proper home on your own server, written for anyone who has priced a hosted automation tool and done the math.

What It Actually Is

Workflow Chain Engine is a visual workflow automation engine you host yourself. You draw a flow in the browser, call an API, branch on the answer, loop over a list, and end in success or fail. The whole system is one small PHP application with a single SQLite file. No framework, no build step, no dependency install, and the editor library ships inside the project, so nothing is loaded from a CDN at runtime.

That constraint is the interesting part. It means the tool is at home on a three dollar shared host, a VPS, or a Docker container, which matters when the process you are automating is worth ten dollars a month rather than two hundred.

The Parts That Matter in Practice

Six node types cover the logic you need: calling out, branching, looping, and terminating in a success or fail state. Variables carry data between steps, so the response from one API call is available to the next node without writing any glue of your own to move it there.

Triggers come in four flavors: a webhook, a schedule, a call from your own code, or a button someone clicks. That last one is underrated. A lot of internal automation is really a task somebody runs by hand at an unpredictable time, and handing that person a button beats handing them a runbook.

The Run Log Is the Feature

Every run leaves a step by step log you can actually read. That sounds mundane right up until you have debugged a hosted automation whose entire error report was the phrase workflow failed. Knowing which node fired, what it received, and what it returned is the difference between a five minute fix and a lost afternoon.

Loop protection is built in as well, which is the other thing you want in place before you point a webhook at something capable of calling itself.

Worth a Look If

You are already self hosting the database and the object storage but still renting the glue. The code is MIT licensed and on GitHub, so the cost of finding out is one afternoon.

From GitHub Repositories to a Professional Data Analytics Portfolio Website

Sat, 08 Aug 2026 07:35:02 +0000

Building More Than Just Projects — Building My Professional Identity

Completing my Data Analyst Internship at Logic Stack was a significant milestone. Over four weeks, I worked on real-world business problems using Excel, Power BI, Python and SQL, successfully completing four practical analytics projects.

Like many learners, I initially thought that uploading my projects to GitHub would be enough.

After all, GitHub is the standard platform for developers and data professionals.

But then I asked myself a simple question:

If a recruiter visits my profile, will they truly understand my journey just by opening four separate repositories?

The answer was No.

That realization inspired me to build something much bigger than another GitHub repository.

It inspired me to build my own Production-Level Data Analytics Portfolio Website.

Why GitHub Alone Wasn't Enough

GitHub is excellent for storing source code and documentation.

However, recruiters usually spend only a few minutes reviewing a candidate's profile.

Opening multiple repositories one by one makes it difficult to understand:

The overall learning journey
Project progression
Skills gained
Business impact
Technical growth

I wanted to create a place where everything could be explored from a single professional dashboard.

The Idea

Instead of sharing multiple independent repositories, I designed a portfolio website that presents my internship as one complete journey.

Rather than saying,

"Here are my projects."

I wanted the website to tell a story:

How I started, what I learned, which challenges I faced, how I solved them and what I built by the end of the internship.

That became the foundation of the entire portfolio.

Designing the Portfolio

The goal was simple:

Create a clean, modern, recruiter-friendly website that highlights both my Data Analytics skills and my Web Development experience.

The portfolio includes carefully designed sections that guide visitors through my internship journey.

Home

A professional landing page introducing my internship, background, and career focus.

Projects

A dedicated section for all four internship projects.

Each project contains:

Project overview
Technologies used
GitHub Repository
LinkedIn post
YouTube explanation video

Everything is connected from one location.

Skills Learned

Instead of simply listing technologies, I organized everything I learned during the internship into one dedicated section.

The portfolio highlights skills including:

Microsoft Excel
Power BI
Python
SQL
Data Cleaning
Exploratory Data Analysis
KPI Development
Business Intelligence
Dashboard Design
Documentation

This gives recruiters a quick overview of my technical capabilities.

Challenges Solved

One of my favourite sections is the Challenges Solved page.

Instead of hiding the problems I encountered, I documented them professionally.

Examples include:

Python environment setup
Library installation issues
SQL query debugging
Data cleaning problems
Documentation challenges
GitHub repository organization

Each challenge is paired with the solution I implemented.

I believe problem-solving demonstrates professional growth more effectively than simply showing successful projects.

Certificate & Internship Documents

The portfolio also includes my internship certificate and professional documents.

To keep them organized, I implemented a simple password-protected section that allows authorized viewers to access these files when needed.

This keeps the homepage clean while still making important documents available.

Building with Modern Technologies

This project also became an opportunity to strengthen my web development skills.

The website was built using modern technologies including:

Next.js
TypeScript
Tailwind CSS
Framer Motion
Responsive Design
SEO Optimization
Structured Data (JSON-LD)
Robots.txt
Sitemap.xml

By combining these technologies, I transformed a simple portfolio into a production-ready web application.

SEO Matters Too

I didn't want the website to look professional only for people who already had the link.

I wanted search engines to understand it as well.

To improve visibility, I implemented:

Metadata optimization
Open Graph tags
Structured Data
Robots configuration
XML Sitemap
Performance optimization

Achieving high Lighthouse scores became another goal during development, ensuring the website follows modern web standards.

Combining Two Different Skill Sets

This project became much more than a portfolio.

It combined two different fields I enjoy:

Data Analytics

Business insights
SQL
Power BI
Python
Excel

Web Development

Next.js
TypeScript
UI Design
Responsive Layouts
SEO
Performance Optimization

Bringing these two disciplines together allowed me to create something that represents my complete technical profile.

Lessons Beyond the Internship

Building this website taught me several important lessons.

A portfolio is not just about displaying work.

It is about presenting your journey professionally.

I learned that:

Presentation matters.
Documentation matters.
User experience matters.
Personal branding matters.
Professional storytelling matters.

A recruiter should be able to understand who you are within minutes.

That became the guiding principle behind this project.

Final Thoughts

Looking back, I'm glad I didn't stop after uploading four repositories.

Creating this portfolio allowed me to organize my internship experience into one professional platform that showcases both my technical skills and my continuous learning journey.

This website represents more than projects.

It represents persistence.

It represents growth.

It represents every challenge solved throughout the internship.

Most importantly, it reminds me that learning never truly ends.

Every project becomes a stepping stone toward the next opportunity.

Explore My Complete Portfolio

🌐 Live Portfolio:
https://yasirawaninfo-data.vercel.app/

💻 GitHub:
https://github.com/YasirAwan4831

Thank you for taking the time to explore my journey. I hope it inspires other learners to build not only projects—but also a professional identity around their work.

#DataAnalytics #SQL #SQLite #PowerBI #Python #Panda #Numpy #Matplotlib #Excel #BusinessIntelligence #DataVisualization #Dashboard #DataCleaning #JupyterNotebook #ProblemSolving #GitHub #WebDevelopment #LearningInPublic #Portfolio #CareerGrowth ##MuhammadYasir #YasirAwan4831

Continuing My Data Analyst Journey: Week 3, 4 and Challenges I Faced During My Data Analyst Intern

Fri, 07 Aug 2026 19:22:22 +0000

In my previous blog, I shared my journey through Week 1 and Week 2, where I worked on Excel-based data cleaning, sales analysis, Pivot Tables and Power BI dashboards.

As the internship progressed, the learning curve became much steeper. The focus shifted from spreadsheet analytics to programming, databases and solving real-world business problems using professional data analysis tools.

Weeks 3 and 4 introduced completely new technologies, larger datasets and more technical challenges that pushed me far beyond my comfort zone.

These two weeks transformed my internship from learning business reporting into building complete end-to-end data analytics solutions.

Week 3: Supply Chain Analytics using Python & Power BI

Week 3 marked my first serious experience with Python for Data Analytics.

Unlike previous weeks, this project required setting up a complete Python development environment before writing a single line of analysis. I worked extensively with Pandas, NumPy, Matplotlib and Jupyter Notebook to clean, analyse and visualize a real-world supply chain dataset containing thousands of records.

After completing the Exploratory Ddata Analysis (EDA), I integrated the insights into Power BI, creating an interactive dashboard that highlighted sales performance, shipment modes, product groups, vendors and country-wise business metrics.

This project taught me that successful data analysis begins with clean, reliable data—and that Python is one of the most powerful tools for transforming raw datasets into meaningful business insights.

GitHub Repository: https://github.com/YasirAwan4831/week-3-python-powerbi-supply-chain-analytics

YouTube Video: https://youtu.be/SQk41XxxpsY?si=ElhxePDt916bh_VJ

LinkedIn Post: https://www.linkedin.com/posts/yasirawan4831_eda-dataabranalyst-fullabrstackabrdevel

Week 4: Funnel & Revenue Analysis using SQL & Power BI

The final week introduced one of the most important skills in modern data analytics: SQL.

For this project, I worked with SQLite to analyse customer funnel performance, revenue generation and conversion metrics directly from a relational database. Instead of relying on spreadsheets, I learned how businesses retrieve insights using SQL queries, aggregate functions, filtering, grouping and KPI calculations.

The extracted data was then connected to Power BI, where I built an interactive business dashboard presenting conversion rates, revenue trends and overall funnel performance.

This project completed my internship journey by combining database querying, business intelligence and visualization into one complete analytics workflow.

GitHub Repository: https://github.com/YasirAwan4831/week-4-sql-powerbi-funnel-analysis

YouTube Video: https://youtu.be/sdBYOKqT6YM?si=HfNw1CrgG71dHIzi

LinkedIn Post: https://www.linkedin.com/posts/yasirawan4831_sqlite-sqlabrbased-dataabranalys

From Learning Tools to Solving Problems

By the end of Week 4, I had worked with:
Microsoft Excel
Power BI
Python
SQL
Business Analytics
Data Cleaning
Exploratory Data Analysis (EDA)
KPI Dashboard Development
Funnel Analysis
GitHub Documentation

However, learning these tools was only part of the journey.
The real growth came from overcoming technical challenges, debugging errors, researching solutions and continuously improving every project.

10 Real Challenges I Faced During My Data Analyst Internship (And How I Solved Them)

Every Internship Teaches Skills — But Real Growth Comes From Solving Problems

When people look at a completed project, they usually see the dashboard, the charts, the SQL queries, or the final GitHub repository.

What they don't see are the countless hours spent fixing errors, learning unfamiliar tools, debugging environments, researching documentation and solving problems that never appeared in any tutorial.

During my one-month Data Analyst Internship at Logic Stack, I completed four industry-based projects covering Excel, Power BI, Python, SQL and Business Analytics.

However, the most valuable outcome wasn't just the projects themselves.

It was learning how to solve problems independently.

This blog shares the biggest technical and practical challenges I faced during my internship—and how each one made me a better Data Analyst.

Week 3 Project

Supply Chain Analytics using Python & Power BI

By Week 3, the internship shifted from Excel into Python-based analytics.

This project focused on cleaning, analysing and visualizing a real-world Supply Chain dataset using Python (Pandas, NumPy, Matplotlib) alongside Power BI.

Unlike previous weeks, this project introduced a completely new development environment, making it both exciting and challenging.

Week 4 Project

Funnel & Revenue Analysis using SQL & Power BI

The final week introduced SQL for business analytics.

The objective was to analyse conversion funnels, revenue performance and customer behaviour using SQL queries before presenting business insights inside Power BI dashboards.

This project simulated real-world reporting workflows where raw database queries become executive dashboards.

Challenge 1

Starting With Zero Practical Experience

Challenge

Although I had experience in web development, I had almost no practical exposure to professional Data Analytics workflows.

Everything—from Excel reporting to SQL analytics—was new.

Solution

Instead of trying to learn everything at once, I followed a structured approach:

Learning one technology at a time helped me build confidence while creating portfolio-ready projects every week.

Challenge 2

Building a Professional Excel Workflow

Challenge

Creating clean spreadsheets is easy.

Creating professional business reports is not.

I had to learn:

Structured Tables
Pivot Tables
Professional Formatting
Multiple Worksheets
Formula Organization

Solution

I researched Excel best practices and redesigned my workbook several times until it followed professional reporting standards.

The result was an organized multi-sheet Excel project that later became the foundation for my Power BI dashboard.

Challenge 3

Learning Power BI Dashboard Design

Challenge

Power BI wasn't difficult because of the software.

It was difficult because dashboards must answer business questions—not simply display charts.

Solution

I learned:

KPI Cards
DAX Basics
Dashboard Layout Principles
Visual Hierarchy
Business Storytelling

Rather than creating colourful visuals, I focused on dashboards that communicate insights clearly.

Challenge 4

Configuring the Python Development Environment

Challenge

This was one of the toughest technical challenges.

Issues included:

VS Code Python interpreter problems
Jupyter Kernel connection failures
Package installation errors
Environment configuration
Version conflicts

These problems had nothing to do with analytics—but they completely blocked development.

Solution

Instead of giving up, I researched official documentation, debugging guides and community discussions.

Eventually I:

Fixed interpreter paths
Installed required libraries
Connected Jupyter successfully
Stabilized the development environment

This experience taught me that environment setup is a real engineering skill.

Challenge 5

Cleaning a Real-World Dataset

Challenge

The Supply Chain dataset contained:

Missing values
Mixed data types
Invalid numeric values
Text inside number columns

Real business datasets are rarely clean.

Solution

Using Pandas, I:

Converted invalid values safely
Filled missing values
Removed inconsistencies
Validated cleaned data

Only after proper cleaning did visualization become meaningful.

Challenge 6

Installing SQLite & Building SQL Workflows

Challenge

Week 4 introduced SQL.

Before writing queries, I first had to configure SQLite and import datasets correctly.

Solution

I installed DB Browser for SQLite, verified database structure, imported CSV datasets correctly and prepared the environment before beginning SQL analysis.

This preparation prevented many future errors.

Challenge 7

Writing Complex SQL Queries

Challenge

Simple SELECT statements were easy.

Business reporting required:

GROUP BY
Aggregate Functions
Filtering
Revenue Calculations
Conversion Metrics

Solution

Instead of memorizing syntax, I practiced writing increasingly complex queries while documenting each command inside my repository.

This improved both my SQL knowledge and documentation skills.

Challenge 8

Turning Data Into Business Insights

Challenge

Anyone can generate numbers.

A Data Analyst must explain what those numbers actually mean.

Solution

For every project I focused on answering business questions:

Which products perform best?
Where is revenue growing?
Which funnel stage loses customers?
What actions should businesses take?

This mindset transformed technical work into business analysis.

Challenge 9

Professional GitHub Documentation

Challenge

Writing code is only half the job.

Presenting projects professionally is equally important.

Solution

I created consistent repositories with:

Detailed README files
Professional documentation
SVG animations
Shields.io badges
Folder organization
Screenshots
Business explanations

These repositories became portfolio-quality projects rather than simple assignments.

Challenge 10

Building My Own Data Analyst Portfolio Website

Challenge

Instead of simply uploading four GitHub repositories, I wanted to create something unique.

Solution

I designed and developed a complete interactive portfolio website using Next.js and TypeScript.

The website includes:

Internship Overview
Four Complete Projects
GitHub Repositories
LinkedIn Posts
YouTube Project Videos
Skills Learned
Challenges Solved
Password-Protected Certificate Section
Resume Download
Modern Responsive UI
SEO Optimization

This project combined my web development background with my newly acquired Data Analytics skills, creating a portfolio that reflects both technical implementation and professional presentation.

What This Internship Really Taught Me

This internship wasn't just about learning Excel, Python, SQL, or Power BI.

It taught me:

How to research independently.
How to debug unfamiliar problems.
How to organize professional projects.
How to document technical work.
How to communicate business insights.
How to keep learning when solutions aren't immediately obvious.

These lessons will remain valuable long after the internship ends.

Final Thoughts

Looking back, the dashboards and repositories represent only the visible outcome.

The real achievement was overcoming every obstacle that appeared along the way.

Every error message, every failed installation, every debugging session and every rewritten query contributed to my growth as a Data Analyst.

The internship has ended—but the learning journey continues.

Thank you, Logic Stack, for providing a practical environment where I could build real projects, solve real problems and gain valuable industry experience.

Connect With Me

🌐 Portfolio
https://yasirawaninfo-data.vercel.app/

💻 GitHub
https://github.com/YasirAwan4831

💼 LinkedIn
https://linkedin.com/in/yasirawan4831

Thank you for reading. If this journey inspires someone starting their own Data Analytics path, then sharing these experiences has been worthwhile.

#DataAnalytics #SQL #SQLite #PowerBI #BusinessIntelligence #DataVisualization #Dashboard #GitHub #Internship #LogicStack #DataAnalyst #LearningJourney #OpenToWork #Portfolio #YasirAwan4831

Expanding the Sovereign AI Stack: Moving the Specification from Gateway to Local Silicon

Tue, 16 Jun 2026 17:34:04 +0000

When I first introduced the Sovereign Systems Specification and released the initial foundation of the SDK, sovereign-core and its accompanying sovereign-fastapi integration layer (see announcement post here), the goal was simple but ambitious: establish a secure, deterministic cryptographic checkpoint at the network ingestion boundary.

sovereign-core gave local infrastructure a way to anchor identity and validate incoming payloads, while sovereign-fastapi provided the high-performance middleware necessary to drop those security primitives cleanly into production web runtimes.

But a secure gateway is only half the battle. As autonomous agents and LLM orchestrators evolve into core enterprise infrastructure, data has to travel deeper into the local topology. It moves across processing loops, through token-minimization filters, and down into persistent storage. If that data isn't armored at every single rest stop, your "sovereign" system still inherits massive operational liabilities.

To move the ecosystem down the road and secure the entire data lifecycle, I am excited to announce the release of the next two core workspace components of the Sovereign SDK: sovereign-sieve and sovereign-ledger.

Together, they transition the stack from a server-side perimeter proxy into a complete, end-to-end local data engineering pipeline.

1. `sovereign-sieve` — Slicing the Prose Tax

Before data can be securely audited, it needs to be optimized. Right now, production AI implementations are burning up to 30% of their cloud compute budgets on what I call the Prose Tax.

sovereign-sieve is an ultra-lightweight, zero-dependency utility that implements our Sieve-and-Sign Pattern.

Instead of routing raw conversational noise directly to downstream agents or databases, sovereign-sieve runs an algorithmic parsing engine locally to clean text streams, isolate underlying data schemas, and strip out fluff. By minimizing your token footprint and context window pressure on local silicon before crossing the ingestion boundary, it turns AI data flow from an unpredictable economic drain into a metered, optimized utility.

Registry: pip install sovereign-sieve
Status: Active & Distributed

2. `sovereign-ledger` — The Immutable Data Vault

Once data has been sieved by the edge and signed by sovereign-core, it requires an un-falsifiable record of custody. Standard application logging is notoriously fragile—anyone with root access or database privileges can alter, backdate, or erase a JSON log file to cover up an algorithmic failure or a security breach.

sovereign-ledger provides a zero-dependency, append-only, SQLite-backed cryptographic audit store engineered specifically for high-concurrency environments.

It enforces the specification's Write-Side Custody mandate through two tightly integrated layers:

Engine-Level SQL Triggers: Compiled directly inside the database file using BEFORE UPDATE and BEFORE DELETE rules that execute a strict RAISE(ROLLBACK, ...). Any mutation attempt from any database client, internal library or external raw connection, is instantly aborted and unwound.
A Linear SHA-256 Hash Chain: Every row is mathematically sealed to its predecessor via an eight-column, NUL-delimited (\x00) canonical preimage. Altering a single timestamp string, tampering with text, or shifting a float precision point out-of-band instantly breaks the chain alignment.

Multi-Writer Concurrency Without Mutex Bloat

To survive asynchronous ASGI web server runtimes (like FastAPI under Uvicorn), sovereign-ledger bypasses slow Python-level mutex locks. Instead, it utilizes threading.local() connection pooling paired with explicit BEGIN IMMEDIATE transaction boundaries.

When multiple concurrent worker threads attempt to write an audit entry, their transactions are cleanly serialized at the SQLite reserved-lock layer, safely queuing inside a 5-second busy_timeout buffer rather than throwing transaction collisions or parent-hash forks.

Registry: pip install sovereign-ledger
Status: Active & Distributed

The Evolving Sovereign Pipeline

By combining these four pieces, the Sovereign SDK now provides a unified, local-first architecture that handles ingestion, minimization, validation, and storage with zero cloud dependencies:

import hashlib
from sovereign_sieve import minimize_payload
from sovereign_ledger import SovereignLedger

# 1. Strip the prose tax via sovereign-sieve
clean_text, metrics = minimize_payload(untrusted_user_input)

# 2. Establish identity and state via sovereign-core / gateway logic
mock_receipt = {
    "payload_hash": hashlib.sha256(clean_text.encode()).hexdigest(),
    "timestamp": "2026-06-16T10:00:00Z",
    "signature": "ecdsa_signature_from_core_gateway",
    "metadata": {
        "prose_tax_summary": metrics
    }
}

# 3. Commit to the immutable vault using sovereign-ledger's context manager
with SovereignLedger(db_path=".keys/audit_trail.db") as ledger:
    # Appends atomically and returns the verified payload identifier
    receipt_id = ledger.append_receipt(mock_receipt, clean_text)
    
    # Run a memory-efficient cursor sweep to verify absolute chain integrity
    assert ledger.verify_ledger_integrity(expected_tip_hash=receipt_id) is True

What’s Next: Expanding to the Edge

With core, fastapi, sieve, and ledger stable, the Sovereign Systems Specification has successfully mapped out the gateway and data storage layers. But to truly complete the lineage of local data, we have to go further downstream. All the way to the exact millisecond data is born.

The next phase of the roadmap will push the boundaries of the SDK out to physical edge silicon:

sovereign-sensor: An ultra-lean cryptographic envelope engine built for MicroPython/CircuitPython (ESP32, Raspberry Pi Pico) to enforce Write-Side Custody at the hardware pin layer.
sovereign-edge: A low-footprint constraint engine optimized for edge compute nodes (Raspberry Pi CM4) to handle structural parsing (§) and offline context snapshots in the field.

The core rule remains unyielding: 100% offline silicon execution, zero telemetry leakages, and absolute dependency minimalism. Check out the new releases, run the adversarial test suites, and let me know how you’re building local-first governance into your production loops.

GitHub Repository: github.com/kenwalger/sovereign-sdk
Sovereign Systems Specification: https://kenwalger.github.io/sovereign-system-spec/

How SQLite Turns Hardware Chaos into Correctness

Sat, 04 Apr 2026 10:30:57 +0000

Hello, I'm Maneshwar. I'm building git-lrc, an AI code reviewer that runs on every commit. It is free, unlimited, and source-available on Github. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.

Databases exist in a hostile world. Disks may fail mid-write. Power cuts interrupt operations. Memory corrupts. Multiple processes access the same file simultaneously.

Yet somehow, SQLite manages to maintain correctness by ensuring committed transactions persist and incomplete ones vanish even when hardware behaves unpredictably.

This post explores how SQLite transforms unreliable hardware into a reliable database system.

It is a concise summary of what I’ve learned over the past month while diving into SQLite’s internals, distilled from a longer ongoing series (Understanding SQLite) and grounded primarily in my study of SQLite Database System: Design and Implementation.

We'll trace the journey from physical storage to the sophisticated mechanisms that guarantee ACID properties, revealing how engineering discipline overcome chaos.

The Foundation: Understanding Hardware Reality

To appreciate SQLite's correctness guarantees, we must first understand what it's working against.

Disks

At the physical level, persistent storage is fundamentally unreliable.

Storage devices expose a byte-addressable interface, but provide no guarantees that multi-byte writes are atomic or durable across power failures.

While filesystems often operate using blocks (commonly 4 KB, but not guaranteed), SQLite cannot rely on any specific block size or atomic-write behavior.

A write operation can fail midway during a power loss, leaving a page only partially updated.

This results in corrupted data that may not be detected by the operating system or hardware.

Consider what happens when a database tries to update data:

If power fails between steps 3 and 4, the data may be lost.

If it fails during step 3, the database page may be partially written, corrupting the database.

The Non-Atomic Write Problem

Database systems cannot ignore this reality. Disk writes are not atomic.

A write operation can fail midway, leaving persistent storage in an inconsistent state. This is why databases:

Use write-ahead logging (WAL)
Avoid overwriting data blindly
Rely on journaling, recovery protocols, and defensive I/O ordering
Treat disks as unreliable, even when hardware claims otherwise

Database designers must explicitly plan for partial writes and silent corruption.

Concurrency: Multiple Processes, One File

SQLite stores an entire database in a single native file.

This design choice heavily influences how concurrency is implemented.

Unlike server databases that can use row-level or table-level locking, SQLite coordinates concurrency using a database-level locking protocol built on top of operating-system file-locking primitives.

Multiple processes may try to:

Read the database simultaneously
Write to different tables at the same time
Modify the same data concurrently

Without proper coordination, these operations would corrupt the database file.

SQLite's Defense Strategy: Layered Correctness

SQLite doesn't fight hardware chaos with complexity.

Instead, it uses a carefully designed set of mechanisms that work together to guarantee correctness:

Journals for crash recovery
File locking for concurrency control
Transaction management for atomicity
Checksums and validation for corruption detection

Let's examine each layer.

Layer 1: Journals: The Safety Net

SQLite's first line of defense against hardware failures is the journal.

A journal is a crash recovery file that records database changes so SQLite can roll back incomplete transactions.

Rollback Journal: Before Images for Recovery

For every database file, SQLite uses a corresponding rollback journal:

Database: emoji-db-v4.db
Journal: emoji-db-v4.db-journal

The rollback journal stores before-images of database pages, complete copies of pages as they existed before modification.

This allows SQLite to restore the database to its previous state if something goes wrong.

How Rollback Journals Work

When a write transaction begins:

SQLite creates a rollback journal file
Before modifying any database page, SQLite writes the entire original page to the journal
SQLite flushes the journal to stable storage
SQLite modifies the database page in memory
When the transaction commits, SQLite finalizes the journal (delete, truncate, or zero the header depending on journal mode)

Journal finalization (after database pages are synced) marks the commit point

If a crash occurs before commit:

The journal file still exists
On next startup, SQLite detects the journal
SQLite validates the journal header and checksums
SQLite reads the before-images from the journal
SQLite restores all modified pages
The database returns to its pre-transaction state

This is physical (age-based) logging: every log record stores the entire database page, even if only one byte is modified.

This approach is intentionally inefficient in space and I/O, but it dramatically simplifies recovery — restoring pages is deterministic and requires no logical replay.

Journal Structure: Segments and Headers

A rollback journal is divided into log segments. Each segment consists of:

A journal header (designed to fit within a single disk sector)
One or more log records, each containing a before-image page

The header begins with eight magic bytes:

D9 D5 05 F9 20 A1 63 D7

These provide sanity checking and help detect torn or invalid writes.

The header also stores:

Number of page records (nRec)
Random seed used for checksum calculations
Original database page count
Disk sector size
Database page size

All values are stored in big-endian format, ensuring cross-platform compatibility.

Statement Journal: Fine-Grained Rollback

A statement journal exists for one specific purpose: undoing the partial effects of a single SQL statement that fails midway, without aborting the surrounding transaction.

Example:

An INSERT statement attempts to insert 1000 rows. If a constraint violation occurs at row 500, the first 499 inserts must be undone, but the user transaction must remain active.

The statement journal:

Is a separate rollback journal file
Stored as a temporary file (random etilqs_* name)
Lives in the system temp directory
Exists only during statement execution
Is deleted immediately after the statement completes

Like rollback journals, statement journals store before-image pages on disk.
However:

They are not used for crash recovery
They are ignored during database startup
They exist only to support statement-level atomicity

SQLite tracks statement journal metadata in memory, allowing it to roll back a single statement without aborting the user transaction.

Master Journal: Multi-Database Atomicity

SQLite allows multiple databases to be attached to a single library connection:

A single transaction may modify multiple databases. Each database has its own rollback journal, but atomicity must be preserved across all of them.

To achieve this, SQLite introduces the master journal.

The master journal:

Is created during commit preparation
Exists only for multi-database write transactions
Is deleted once commit completes
Contains no page images
Stores the names of participating rollback journals

Each participating rollback journal becomes a child journal. During commit, each child journal appends a master journal record containing.

If SQLite crashes during commit:

Recovery examines child journals
Child journals reference the master journal
The master journal confirms which journals belong together
Either all databases are rolled back, or all are committed

This guarantees global atomicity across attached databases.

Layer 2: File Locking

Journals protect against crashes, but they don't solve concurrency.

SQLite relies on file locking to coordinate access across processes.

DB Level Locking Model (Rollback-Journal Mode)

In rollback-journal mode, SQLite enforces strict rules:

Multiple readers may read concurrently
Only one writer may exist at a time
Readers block writers
Writers block both readers and other writers

This coarse-grained locking model simplifies correctness and eliminates deadlocks, at the cost of reduced concurrency.

Lock States and Transitions

SQLite uses five lock states:

UNLOCKED: No locks held
SHARED: Multiple readers allowed
RESERVED: Writer intends to write
PENDING: Writer waiting for readers to finish
EXCLUSIVE: Writer actively modifying db

Typical transitions:

Readers acquire SHARED
Writers acquire SHARED → RESERVED → PENDING → EXCLUSIVE
Write locks are released at transaction end
Read locks may be released earlier, depending on execution flow

Continue reading the rest of article

here

My experiments and hands-on executions related to SQLite will live here: lovestaco/sqlite

References:

SQLite Database System: Design and Implementation. N.p.: Sibsankar Haldar, (n.d.).

SQLite - High-Level Introduction

Tue, 24 Jun 2025 15:27:25 +0000

SQLite

SQLite is a self-contained, serverless, zero-configuration relational database engine. Unlike traditional database servers, SQLite is distributed as a simple library that you embed directly into your applications. This guide will introduce you to SQLite's core concepts, explain why it's such a popular choice for developers, highlight how it differs from other database systems, and show how you can leverage it inside Divooka's visual programming environment.

Why SQLite?

For many new programmers, setting up a client–server database (e.g., MySQL, PostgreSQL) can feel like learning two things at once: programming and system administration. SQLite removes that overhead:

Zero-configuration
No daemon to install or manage - your application just opens a file.
Lightweight and portable
Entire database lives in a single cross-platform file (often just a few hundred kilobytes of library code).
Broad language support
Bindings exist for Python, JavaScript, C#, Java, Go, Rust, and more - so you can start querying data in the language you already know.

Because of its simplicity, SQLite is the most widely deployed database engine in the world, powering everything from mobile apps and desktop software to IoT devices and web browsers.

Core Architecture

File-Based Storage

Single file: Definitions, tables, indexes, and data all live in one .db (or .sqlite3) file.
Cross-platform: You can copy the database file between Windows, macOS, Linux, and mobile devices without conversion.

The file-based nature of SQLite makes it super easy to backup and migrate databases.

Serverless and Embedded

Library, not a server: Applications link against libsqlite3 and call C-APIs (or higher-level wrappers) directly - there's no separate "database process."
Reduced latency: Reads and writes happen in-process, avoiding inter-process communication overhead found in client–server setups.

Dynamic Typing ("Manifest Typing")

Types on values, not columns: Columns have "type affinities" (e.g., INTEGER, TEXT), but SQLite won't reject a string inserted into an INTEGER column - it will try to convert it.
Optional strict mode: As of recent versions, you can declare CREATE TABLE ... STRICT to enforce traditional SQL type checking if you need it.

Concurrency Model

Reader–writer locks: Multiple readers can operate simultaneously, but writes acquire an exclusive lock on the file.
Write-Ahead Logging (WAL): Enables concurrent reads and writes by appending changes to a separate log, improving throughput for mixed workloads.

Key Differences from Other Databases

Aspect	SQLite	Client–Server Databases
Deployment	Embed a library; open a local file	Install/maintain a separate server
Configuration	Zero-config; managed by file system	Requires user accounts, ports, etc.
Process architecture	In-process library calls	Inter-process with network protocol
Typing	Dynamic, manifest typing	Static, enforced column types
Concurrency	Serialized writes (with WAL option)	True concurrent writes via server
Backup	Copy the file or use online backup APIs	Use dump utilities or replication

Practical Use Cases

Desktop and Mobile Apps
Store user settings, preferences, and offline data without bundling a full database server.
Embedded and IoT Devices
Firmware and edge-computing modules often rely on SQLite for local data persistence, thanks to its tiny footprint.
Prototyping and Testing
Quickly spin up a relational store for experiments and unit tests - no setup scripts needed.
High-Availability Reads
Applications that require many concurrent read operations (e.g., analytics dashboards) benefit from SQLite's low-latency access.
File-Based Data Interchange
Sharing a .db file between teams or tools (e.g., GIS software, data science notebooks) is as simple as sharing a CSV.

SQLite in Divooka

In Divooka's visual programming language, SQLite plays two roles:

In-Memory Data Engine: Run queries against data you're manipulating on the canvas - think temporary tables without disk I/O.
Persistent Storage: Divooka Explore ships with SQLite pre-installed, so you can save and load your data without any extra configuration - just specifying a filepath for the database and you are good to go.
Simple Web Interface: The O'ista web framework offers a capability for you to host SQLite online with authentication, so the database can be accessed by others.

This integration means you can drag a "SQL Query" node onto your workflow, point it at either an in-memory dataset or a saved .db file, and immediately start filtering, aggregating, and joining tables - all within Divooka's visual interface.

Getting Started

Open/Create a database

-- In the sqlite3 shell:
.open my_project.db

Create tables

CREATE TABLE users (
  id INTEGER PRIMARY KEY,
  name TEXT NOT NULL,
  joined DATE
);

Insert sample data

INSERT INTO users (name, joined)
VALUES ('Alice', '2025-06-01'),
       ('Bob',   '2025-06-15');

Run a query

SELECT id, name
FROM users
WHERE joined >= '2025-06-10';

In Divooka Explore, these same SQL snippets can be entered into a SQL query block node, letting you mix SQL with visual data flows.

Conclusion

SQLite offers a frictionless path for beginners to learn SQL and build data-driven applications, while still providing advanced features like ACID transactions, full-text search, and JSON support for more experienced developers. Its unique combination of simplicity, portability, and performance makes it an ideal companion for both standalone projects and integrations inside environments like Divooka. Whether you're just exploring databases for the first time or architecting an embedded system, SQLite is a tool that scales with your needs - no server required.

References

Database Interaction using Python: Introduction to SQLite/MySQL/PostgreSQL

Wed, 19 Mar 2025 11:56:30 +0000

The article covers POSTGRESQL, MySQL, and SQLITE3—three major database management systems. They provide unique features and benefits and are essential to software development. Let's explore their features so that you can make better selections that will improve the project. Together, let's explore these essential elements in modern software engineering.

As SQLite is lightweight, it is a great choice for embedded applications and situations where simplicity is essential. On the other hand, MySQL stands out thanks to its strong client-server design, scalability, and widespread industry acceptance. PostgreSQL, on the other hand, is a popular option for enterprise-level applications due to its versatility, dependability, and compliance with ACID short for Atomicity, Consistency, Isolation, and Durability.

SQLite Database

SQLite is a relational database engine that is lightweight, serverless, self-contained, and requires no configuration. Because it works directly with files, it's perfect for low-resource applications. Distribution and deployment are made easier by the fact that its self-contained databases are kept as single files.

Features:

Embedded in applications: SQLite, which is frequently integrated into programmes, allows internal data administration without the need for external servers. Data management is made easier by this, especially for offline functioning.
File-based database: SQLite databases are manageable and portable since they are kept as individual files on the file system.
Appropriate for small to medium-sized applications: Because of its simplicity and lightweight design, SQLite is especially well-suited for small to medium-sized applications, even though it can manage huge datasets as well.

Let's go through an example of defining tables, installing SQLite in Python, and carrying out CRUD (Create, Read, Update, Delete) operations:

conn = sqlite3.connect('example.db')

cursor = conn.cursor()

cursor.execute('''CREATE TABLE IF NOT EXISTS users

                  (id INTEGER PRIMARY KEY, name TEXT, email TEXT)''')

cursor.execute("INSERT INTO users (name, email) VALUES (?, ?)", ('Alice', '*Emails are not allowed*'))

cursor.execute("INSERT INTO users (name, email) VALUES (?, ?)", ('Ali', '*Emails are not allowed*'))

print("Records after initial insertion:")

cursor.execute("SELECT DISTINCT * FROM users")

rows = cursor.fetchall()

for row in rows:

    print(row)

cursor.execute("UPDATE users SET email = ? WHERE name = ?", ('*Emails are not allowed*', 'Alice'))

print("\nRecords after updating email of Alice:")

cursor.execute("SELECT DISTINCT * FROM users")

rows = cursor.fetchall()

for row in rows:

    print(row)

cursor.execute("DELETE FROM users WHERE name = ?", ('Alice',))

print("\nRecords after deleting Alice:")

cursor.execute("SELECT DISTINCT * FROM users")

rows = cursor.fetchall()

for row in rows:

    print(row)

Output:

Tip: Make sure the 'DISTINCT' keyword is applied to the right columns in SQL queries to obtain unique value combinations and prevent accidental duplicate results.

MySQL Database

The popular open-source RDBMS MySQL provides strong features and dependability for systems ranging from small-scale initiatives to enterprise-level setups. It offers a wide range of tools and features for effective structured data management.

Features:

Client-server architecture:MySQL has a client-server design, which enables several clients to connect to the database server at once. The management of database resources, scalability, and security are all improved by this architecture.
Performance and scalability: MySQL is designed to be scalable, meaning it can manage big databases and high transaction volumes while maintaining optimal performance. Performance is improved by its effective caching, query optimisation, and indexing methods.
Broad industrial adoption: MySQL is widely used in a number of industries, including telecommunications, e-commerce, technology, and finance. Its widespread community support, dependability, and open-source nature have all added to its appeal.

You can get the MySQL Community Server package from mysql official website and install MySQL by following the given steps. As an alternative, you can use package managers such as Homebrew for macOS or apt-get for Linux.

Connecting to MySQL from Python:

import mysql.connector

conn = mysql.connector.connect(user='user', password='password',

                               host='localhost',

                               database='test')

cursor = conn.cursor()

cursor.execute("SELECT * FROM table_name")

results = cursor.fetchall()

conn.close()

Creating tables and executing queries using Python's MySQL Connector:

cursor = conn.cursor()

cursor.execute('''CREATE TABLE IF NOT EXISTS users

                  (id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(255), email VARCHAR(255))''')

cursor.execute("INSERT INTO users (name, email) VALUES (%s, %s)", ('Alice', '*Emails are not allowed*'))

cursor.execute("SELECT * FROM users")

results = cursor.fetchall()

PostgreSQL Database

Strong and open-source, PostgreSQL is an object-relational database system that is well-known for its feature-rich set, dependability, and extensibility. It is a top option for both small-scale applications and large corporate systems because of its comprehensive functions that enable complicated data types, transactions, and data integrity.

Features:

Extensibility: Because of PostgreSQL's flexible architecture, programmers are able to design unique functions, data types, and indexing techniques to meet the needs of particular applications. The smooth integration of third-party extensions and libraries is made possible by this flexibility.
ACID Compliance: PostgreSQL guarantees ACID (
Atomicity: To preserve database integrity, transactions are either fully completed or entirely aborted.
Consistency: Assures that the database is kept intact by writing only legitimate data.
Isolation: Data integrity is ensured by many transactions running simultaneously without interfering.
Durability: Even in the event of a system failure, committed transaction modifications remain intact.) compliance, which in turn provides dependable and consistent database transactions. This functionality is crucial for ensuring consistency across the database and data integrity even in the case of system failures.
Data Integrity and Reliability: PostgreSQL uses built-in restrictions like triggers and rules, foreign keys, and concurrency management to guarantee data integrity and stability.

To install PostgreSQL, you can download and install the PostgreSQL package from the official website.

import psycopg2

conn = psycopg2.connect(

    dbname='your_database_name',

    user='your_username',

    password='your_password',

    host='localhost',

    port='5432'

)

if conn:

    print("Connected to PostgreSQL")

else:

    print("Failed to connect to PostgreSQL")

conn.close()

Creating tables and executing queries using Python's psycopg2 library:

# Create a cursor object

cursor = conn.cursor()

cursor.execute('''CREATE TABLE IF NOT EXISTS users (

                    id SERIAL PRIMARY KEY,

                    name VARCHAR(255),

                    email VARCHAR(255)

                )''')

cursor.execute("INSERT INTO users (name, email) VALUES (%s, %s)", ('Alice', '*Emails are not allowed*'))

cursor.execute("SELECT * FROM users")

results = cursor.fetchall()

for row in results:

    print(row)

conn.commit()

Note: Always securely handle database credentials and avoid hardcoding them directly into your code for better security.

Comparison

Factor	SQLite	MySQL	PostgreSQL
Performance	Good for small to medium -sized applications	Optimized for read-heavy workloads	Optimized for complex queries and large datasets
Scalability	Limited scalability due to file- based architecture	Good scalability with support for clustering and sharding	High scalability with advanced replication and partitioning
Features	Basic features with limited concurrency support	Rich feature set including stored procedures, triggers, and views	Comprehensive feature set including JSONB data type, full-text search, and advanced indexing
Ease of Use	Simple setup and usage, suitable for beginners	Easy to use, especially with graphical tools like MySQL Workbench	Requires more configuration but offers powerful features
Use Case Suitability	Ideal for embedded systems and small applications	Well-suited for web applications and medium- sized databases	Best for large-scale applications with complex data requirements

Advanced Concepts

Advanced ideas like indexing, optimization, and transactions are essential for improving the scalability, performance, and dependability of databases.

Transactions: Transactions ensure the ACID properties of database operations, handling multiple tasks as a single unit for data accuracy and reliability. All database systems provide tools for effective transaction management.
Indexing: Indexes expedite data retrieval, particularly for SELECT queries, by swiftly accessing entries in database tables based on specific field values. Effective indexing reduces the need for full table scans, significantly enhancing query performance.
Optimization: Database optimization strategies improve effectiveness and performance through resource usage optimization, performance tuning, query optimization, and schema design optimization. Each system provides unique features and tools to enhance productivity and address performance issues.

import sqlite3

import mysql.connector

import psycopg2

def sqlite_transactions():

    conn = sqlite3.connect('example.db')

    cursor = conn.cursor()

    try:

        conn.execute("BEGIN")

        cursor.execute("UPDATE users SET balance = balance - 100 WHERE id = 1")

        cursor.execute("UPDATE users SET balance = balance + 100 WHERE id = 2")

        conn.commit()

        print("Transaction committed successfully")

    except sqlite3.Error as e:

        conn.rollback()

        print("Transaction rolled back due to error:", e)

    finally:

        conn.close()

def mysql_transactions():

    conn = mysql.connector.connect(user='user', password='password', database='example')

    cursor = conn.cursor()

    try:

        conn.start_transaction()

        cursor.execute("UPDATE users SET balance = balance - 100 WHERE id = 1")

        cursor.execute("UPDATE users SET balance = balance + 100 WHERE id = 2")

        conn.commit()

        print("Transaction committed successfully")

    except mysql.connector.Error as e:

        conn.rollback()

        print("Transaction rolled back due to error:", e)

    finally:

        conn.close()

def postgres_transactions():

    conn = psycopg2.connect(dbname='example', user='user', password='password', host='localhost')

    cursor = conn.cursor()

    try:

        conn.autocommit = False

        cursor.execute("UPDATE users SET balance = balance - 100 WHERE id = 1")

        cursor.execute("UPDATE users SET balance = balance + 100 WHERE id = 2")

        conn.commit()

        print("Transaction committed successfully")

    except psycopg2.Error as e:

        conn.rollback()

        print("Transaction rolled back due to error:", e)

    finally:

        conn.close()

sqlite_transactions()

mysql_transactions()

postgres_transactions()

SQLite, designed for single-user and small-scale applications, uses local file-based databases and simple transaction management with BEGIN, COMMIT, and ROLLBACK statements.
MySQL, a robust RDBMS, offers scalability and concurrency control with start_transaction(), commit(), and rollback() methods.
PostgreSQL, known for advanced features and extensibility, provides strong support for concurrency, data integrity, and advanced SQL features, managing transactions via autocommit mode and commit()/rollback() methods.

FAQ
Q: What is the difference between SQL and NoSQL databases?
A: SQL (Structured Query Language) databases, also known as relational databases, organize data into tables with predefined schemas and use SQL for querying. NoSQL databases, on the other hand, store data in flexible, non-tabular formats and do not require a fixed schema. They are designed for scalability and handling unstructured data.

Conclusion

Databases are essential to software development because they allow for effective data management. MySQL scales for the web, PostgreSQL meets enterprise needs, and SQLite is ideal for lightweight applications. When choosing a project, it is essential to comprehend their strengths. Using cutting-edge ideas promotes creativity in digital solutions by ensuring data performance and dependability.

Reference

For more reference:
SQLite Official Website:
SQLite Official Website

Coder Legion - Recent posts tagged sqlite

Offline-First Bible Reader on iOS: Bundled SQLite, Full-Text Search and iCloud-Synced Notes

1. The problem: two kinds of data with opposite needs

2. A verse ID that will never change

3. Shipping the text as a bundled, read-only SQLite database

4. Search that feels instant

5. User data: Core Data with CloudKit mirroring

6. Listen mode with AVSpeechSynthesizer

7. Verse of the Day on the Home Screen with WidgetKit

8. Real-world example: Bible Study AI – Read Offline

9. Before you ship an offline-first reader: a checklist

Wrapping up

SQLite's Two-Person Team Just Passed Its First AI Stress Test

Alembic in Practice: Structure, Revisions, and the Traps SQLite Sets

The layout

Why the filenames look like keyboard mashing

Why not 0001, 0002, 0003?

If you want readable filenames

The configuration trap that cost two months

Why nothing broke for so long

The fix: one owner for the URL

Autogenerate, and where to stop trusting it

Trap: SQLite cannot ALTER a constraint

Attempt 1: silently kept the old constraint

Attempt 2: copy_from to suppress reflection

Attempt 3: the indexes were gone

Downgrades that refuse to run

A simple migration, for contrast

Keeping constraints from drifting

Testing migrations

Command reference

Checklist

Shrinking a Materials Science Dataset

Ranking New Papers By Meaning Instead Of Keywords, In One PHP App

Why Keyword Alerts Keep Missing The Right Paper

The Shape Of The Pipeline

Choosing Where The Ranking Runs

What Transfers To Your Own Watcher

One PHP App and One SQLite File: A Self Hosted Workflow Engine

What It Actually Is

The Parts That Matter in Practice

The Run Log Is the Feature

Worth a Look If

From GitHub Repositories to a Professional Data Analytics Portfolio Website

Building More Than Just Projects — Building My Professional Identity

Why GitHub Alone Wasn't Enough

The Idea

Designing the Portfolio

Home

Projects

Skills Learned

Challenges Solved

Certificate & Internship Documents

Building with Modern Technologies

SEO Matters Too

Combining Two Different Skill Sets

Data Analytics

Web Development

Lessons Beyond the Internship

Final Thoughts

Explore My Complete Portfolio

Continuing My Data Analyst Journey: Week 3, 4 and Challenges I Faced During My Data Analyst Intern

Week 3: Supply Chain Analytics using Python & Power BI

Week 4: Funnel & Revenue Analysis using SQL & Power BI

From Learning Tools to Solving Problems

10 Real Challenges I Faced During My Data Analyst Internship (And How I Solved Them)

Week 3 Project

Supply Chain Analytics using Python & Power BI

Week 4 Project

Funnel & Revenue Analysis using SQL & Power BI

Challenge 1

Starting With Zero Practical Experience

Challenge

Solution

Challenge 2

Building a Professional Excel Workflow

Challenge

Solution

Challenge 3

Learning Power BI Dashboard Design

Attempt 2: `copy_from` to suppress reflection

1. `sovereign-sieve` — Slicing the Prose Tax

2. `sovereign-ledger` — The Immutable Data Vault