Coder Legion - Recent posts tagged mongodb

Your Pagination Is Showing Users the Same Item Twice (and Skipping Others). Here's Why Offset Breaks

Mon, 26 Oct 2026 11:11:24 +0000

A user scrolls through your feed. Page 1 ends with "Post 20". They tap "Next", and page 2 starts with... "Post 20" again.

Another user pages through an admin table and never sees order #4817. It exists. It is in the database. It just never showed up on any page.

Nobody changed any code. The data changed underneath the pagination, and your skip and limit could not cope.

The Code Everyone Writes

app.get('/api/posts', async (req, res) => {
  const page = Number(req.query.page) || 1;
  const limit = 20;

  const posts = await Post.find()
    .sort({ createdAt: -1 })
    .skip((page - 1) * limit)
    .limit(limit);

  res.json(posts);
});

It is simple, it works in development, and ?page=2 is easy to understand. That is why it is everywhere.

Why It Breaks

Offset pagination says "skip the first N rows, then give me the next 20". It counts positions, and positions move when the data does.

Duplicates. Someone publishes a new post while a reader is on page 1. Every existing post shifts down one position. When the reader requests page 2, the last item of page 1 is now the first item of page 2.

Skips. The reverse: a post is deleted while the reader is on page 1. Everything shifts up one, and the first item of what would have been page 2 slides into page 1's range. The reader never sees it.

Before:  [1 2 3 4 5 | 6 7 8 9 10]    page 1 = 1-5, page 2 = 6-10
New item arrives at the top:
After:   [0 1 2 3 4 | 5 6 7 8 9]     page 2 now starts with 5, which they already saw

The Second Problem: It Gets Slower on Deep Pages

skip(100000) does not jump to row 100,000. The database has to walk past 100,000 documents and throw them away before returning your 20. Page 1 is fast. Page 5,000 is slow, and the cost grows with the page number.

This is why a feed that feels fine with 1,000 rows slows down quietly at a million.

The Fix: Cursor Pagination

Instead of "skip N", say "give me the 20 items after this one". The anchor is a cursor: a value that identifies where the last page ended.

A new item arriving at the top cannot shift your place, because you are not counting positions. You are standing on a specific item.

Step 1: Sort by something stable, with a tie-breaker

createdAt alone is not enough, because two documents can share the same timestamp. Add _id as a tie-breaker so the order is always exact:

.sort({ createdAt: -1, _id: -1 })

Step 2: Support the query with an index

PostSchema.index({ createdAt: -1, _id: -1 });

With this index, the database jumps straight to the cursor and reads 20 documents, whether you are on page 1 or page 5,000.

Step 3: Query "after the cursor"

app.get('/api/posts', async (req, res) => {
  const limit = 20;
  const cursor = req.query.cursor ? decodeCursor(req.query.cursor) : null;

  const filter = cursor
    ? {
        $or: [
          { createdAt: { $lt: cursor.createdAt } },
          { createdAt: cursor.createdAt, _id: { $lt: cursor._id } },
        ],
      }
    : {};

  const rows = await Post.find(filter)
    .sort({ createdAt: -1, _id: -1 })
    .limit(limit + 1); // fetch one extra to know if there is a next page

  const hasMore = rows.length > limit;
  const items = hasMore ? rows.slice(0, limit) : rows;
  const last = items[items.length - 1];

  res.json({
    items,
    nextCursor: hasMore ? encodeCursor(last) : null,
  });
});

The $or says: "older than the cursor, or the same timestamp but a smaller _id". That second branch is what stops items with identical timestamps from being skipped or repeated.

Step 4: Encode the cursor so clients do not depend on it

const encodeCursor = (doc) =>
  Buffer.from(JSON.stringify({ createdAt: doc.createdAt, _id: doc._id })).toString('base64url');

const decodeCursor = (str) => {
  const { createdAt, _id } = JSON.parse(Buffer.from(str, 'base64url').toString());
  return { createdAt: new Date(createdAt), _id: new mongoose.Types.ObjectId(_id) };
};

Clients treat it as an opaque string and send it back as it is. You can change what is inside later without breaking anyone. In production, validate the decoded values and handle a malformed cursor with a 400, not a crash.

What You Give Up

Cursor pagination has a trade-off, and you should know it before switching:

No "jump to page 47". You can only go forward (or back, with a second cursor). For feeds and infinite scroll, that is perfect. For a numbered page bar, it is not.
No cheap total count. countDocuments on a large collection is not free either way, so many apps drop the "page 3 of 500" display.

When Offset Is Still Fine

Do not rewrite everything. Offset pagination is a reasonable choice for small, slow-changing tables, like an admin list with a few hundred rows, where users really do want page numbers.

Situation	Use
Feed, timeline, infinite scroll	Cursor
Large collection, deep pages	Cursor
Data changes while people browse	Cursor
Small admin table with page numbers	Offset is fine

The Rule

Offset pagination counts positions, and positions change. Cursor pagination anchors to a specific item, so it stays correct when data is added or removed. Always sort by a unique, indexed combination, and fetch one extra row to know whether there is a next page.

Quick audit: search your project for .skip(. For each hit, ask whether the collection can grow past a few thousand documents, or change while someone is paging through it.

If you work with MongoDB, I collect more patterns like this, around indexes and queries, in my MongoDB and Mongoose patterns guide, and you can browse my Next.js templates on pixelanas.com.

Get the templates: https://pixelanas.gumroad.com

Have you ever been bitten by duplicate or missing items in a paginated list? Tell me how you noticed in the comments.

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

Your 'Pay Now' Button Can Charge Twice. Idempotency Keys Are the Fix Nobody Adds Until It Happens

Sun, 25 Oct 2026 11:09:21 +0000

A customer clicks "Pay now". Nothing happens for three seconds, so they click again. Or their phone loses signal for a moment and the app quietly retries the request. Or your load balancer times out and sends the request again.

Your server received the same order twice, and it did exactly what you told it to do: it created two orders.

This is not a rare edge case. It is the default behavior of every POST endpoint you have ever written.

The Endpoint That Looks Fine

app.post('/api/orders', requireAuth, async (req, res) => {
  const order = await createOrder(req.user.id, req.body);
  await chargeCard(order);
  res.status(201).json({ orderId: order.id });
});

Nothing is wrong with this code in isolation. The problem is what it cannot know: whether this request is new, or a repeat of one that already succeeded.

Why This Is Hard to Catch

You test with one request at a time. The bug needs two requests with the same intent, close together.
Timeouts are ambiguous. When a client times out, it does not know if the server never got the request, or did the work and the response got lost. Retrying is the only safe-looking choice, and it is the one that duplicates.
Disabling the button is not enough. It helps with double clicks but not with network retries, mobile apps, or another service calling your API.

The Idea: Same Key, Same Result

An idempotent operation is one where doing it twice has the same effect as doing it once. POST is not idempotent by default, but you can make it behave that way:

The client generates a unique idempotency key for each intent and sends it in a header.
The server remembers the key together with the result.
If the same key arrives again, the server returns the saved result and does not do the work again.

Payment providers such as Stripe use exactly this pattern with an Idempotency-Key header.

A Working Version With Express and MongoDB

First, a unique index, plus an expiry so old keys clean themselves up:

await keys.createIndex({ userId: 1, key: 1 }, { unique: true });
await keys.createIndex({ createdAt: 1 }, { expireAfterSeconds: 86400 });

Then the endpoint:

app.post('/api/orders', requireAuth, async (req, res) => {
  const key = req.get('Idempotency-Key');
  if (!key) return res.status(400).json({ error: 'Idempotency-Key header required' });

  const userId = req.user.id;
  const bodyHash = hash(stableStringify(req.body));

  // 1. Try to claim the key. Only one request can win this insert.
  try {
    await keys.insertOne({ userId, key, bodyHash, status: 'processing', createdAt: new Date() });
  } catch (err) {
    if (err.code !== 11000) throw err; // 11000 = duplicate key

    const saved = await keys.findOne({ userId, key });
    if (saved.bodyHash !== bodyHash) {
      return res.status(422).json({ error: 'Key was used with a different request' });
    }
    if (saved.status === 'processing') {
      return res.status(409).json({ error: 'Request is still being processed' });
    }
    return res.status(saved.statusCode).json(saved.response); // replay
  }

  // 2. We own the key. Do the work once.
  try {
    const order = await createOrder(userId, req.body);
    await chargeCard(order);

    const response = { orderId: order.id };
    await keys.updateOne(
      { userId, key },
      { $set: { status: 'done', statusCode: 201, response } }
    );
    return res.status(201).json(response);
  } catch (err) {
    await keys.deleteOne({ userId, key }); // let the client retry
    throw err;
  }
});

The Race Condition Most Examples Miss

A common first attempt is:

const existing = await keys.findOne({ userId, key });
if (existing) return res.json(existing.response);
// ... do the work ...
await keys.insertOne({ userId, key, response });

Two simultaneous requests both run findOne, both see nothing, and both do the work. You added a check and still got a duplicate.

The fix is in the version above: claim the key first with an insert that can only succeed once, and let the unique index decide the winner. The database settles the race, not your code.

Four Details That Matter

1. Scope keys per user. Two users who happen to generate the same key must not see each other's responses. That is why the index is on { userId, key }.

2. Compare the request body. If the same key arrives with a different body, that is a bug or an attack, not a retry. Return an error instead of replaying a response that does not match the request. Use a stable stringify (sorted keys) for the hash, or two identical bodies with different property order will hash differently.

3. Generate the key per intent, not per attempt. On the client, create the key when the user starts the action and reuse it for every retry of that action. A new key per retry brings the duplicate right back.

const key = crypto.randomUUID(); // once, when the checkout starts
await fetch('/api/orders', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'Idempotency-Key': key },
  body: JSON.stringify(cart),
});

4. Pass the key on to your payment provider too. If your server crashes after charging the card but before saving the result, your own table cannot help. A key sent to the provider protects that gap, and the expiry index clears any stuck processing record.

The Rule

Any request that creates something or moves money must be safe to send twice. Do not rely on disabled buttons or "the network is reliable". Give the client a key, claim it atomically on the server, and replay the saved result on a repeat.

Quick audit: list your POST endpoints that create orders, payments, invoices, emails or bookings. For each, ask what happens if the request arrives twice, a second apart. If the answer is "two of them", you have found your next ticket.

If you work with MongoDB, my MongoDB and Mongoose patterns guide has more on using the database to enforce rules, and you can browse my Next.js templates on pixelanas.com.

Get the templates: https://pixelanas.gumroad.com

Have you ever shipped a double-charge or a duplicate-order bug? Tell me how you found out, in the comments.

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

Demystifying MongoDB Write Concern and Read Concern in Node.js

Mon, 19 Oct 2026 09:22:10 +0000

What Are Write Concern and Read Concern?

In MongoDB, write concern and read concern are configuration settings that specify the level of guarantee you require when saving or retrieving data across a distributed cluster. A write concern determines how many database nodes must successfully record an update—and whether that data must be written to disk—before MongoDB reports success to your application. A read concern determines how fresh and consistent the data returned by a query must be, protecting your application from reading temporary data that might later be reversed.

Together, these two settings let developers choose where their application sits on the classic spectrum between maximum execution speed and rock-solid data safety.

The Real-World Analogy: Certified Mail vs. Kitchen Whiteboards

Imagine you run a bustling catering kitchen with multiple head chefs.

When taking an order, you have two choices for your write concern. You could scribble the customer's allergy request on a loose scrap of paper and immediately yell "Order placed!" (a fast write, but if you drop the paper, the order is lost). Alternatively, you could walk the order ticket over to the main order board, watch two sous-chefs sign off on it, and wait for them to file it in the metal lockbox before confirming the order with the customer. The second route takes several seconds longer, but you are guaranteed that the kitchen will never forget the order even if one chef walks out.

Now consider the read concern. If a waiter needs to know the soup of the day, they can either glance at a whiteboard where someone started drafting ideas with an erasable marker, or they can refuse to serve anything until they see the official menu stamped and approved by the head kitchen manager. Glancing at the draft whiteboard is instantaneous, but you might serve an experimental soup that the manager ultimately rejects five minutes later.

Why It Matters in Daily Software Engineering

Modern web architectures rarely run on a single database server. Instead, they use replica sets—groups of connected servers where one "primary" node handles writes and multiple "secondary" nodes replicate that data for redundancy.

If you leave MongoDB on default settings during high-traffic moments, subtle race conditions emerge:

Rollback Catastrophes: If an API endpoint confirms an e-commerce order after writing only to the primary node, and that primary node suddenly experiences a hardware failure before syncing to secondary nodes, the cluster elects a new primary. Your customer's order is wiped out, yet your payment processor already charged their credit card.
Dirty Reads: A background worker updates a bank balance. A microservice queries the secondary replica before the primary finishes replicating. If that transaction aborts or fails over, the microservice has operated on data that never officially existed.

By tuning write and read concerns, software engineers intentionally shield critical operations (payments, account transfers, credential updates) from data loss while letting low-risk operations (logging, view counters, user analytics) run with blazing speed.

Implementation in Node.js

Here is how you apply write and read concerns when working with the official MongoDB Node.js driver:

import { MongoClient } from 'mongodb';

const client = new MongoClient('mongodb://localhost:27017,localhost:27018/?replicaSet=rs0');

async function processPayment(orderId, paymentDetails) {
  await client.connect();
  const db = client.db('store');

  // Configure high-durability options for financial transactions
  const ordersCollection = db.collection('orders', {
    writeConcern: {
      w: 'majority',      // Must be acknowledged by more than half of the replica set nodes
      j: true,            // Must be written to the on-disk journal before confirming
      wtimeout: 5000      // Abort if replicas do not respond within 5 seconds
    },
    readConcern: {
      level: 'majority'   // Only read data committed by a majority of nodes
    }
  });

  // Step 1: Safely write with acknowledgment
  const writeResult = await ordersCollection.updateOne(
    { _id: orderId },
    { $set: { status: 'PAID', payment: paymentDetails } }
  );

  // Step 2: Read guaranteed, durable data
  const confirmedOrder = await ordersCollection.findOne({ _id: orderId });

  console.log('Payment processed safely:', confirmedOrder._id);
  return confirmedOrder;
}

In this example, specifying w: 'majority' and j: true ensures that the write cannot be rolled back if the primary server abruptly crashes. Setting level: 'majority' on the read ensures your application reads only finalized records.

Key Takeaway

Database speed and data reliability exist on a continuous slider: write concern dictates how hard your cluster works to preserve your updates, while read concern dictates how strictly your cluster verifies the stability of what you fetch. Mastering both allows you to protect financial transactions and vital data without unnecessarily throttling high-volume, low-risk endpoints across your stack.

Enjoyed this breakdown? Support my open-source work on Buy Me a Coffee or explore react-hook-lab!

Originally published on my blog. You can read the alternative breakdown here.

Your Mongoose pre('save') Hook Is Being Silently Skipped by Half Your Update Calls

Tue, 13 Oct 2026 09:40:18 +0000

This is one of the more consequential Mongoose gotchas because of exactly what tends to live inside a pre('save') hook, password hashing, slug generation, validation logic, and exactly which methods silently bypass it without ever telling you.

The Setup

// models/User.ts
const userSchema = new Schema({
  email: String,
  password: String,
});

userSchema.pre('save', async function (next) {
  if (this.isModified('password')) {
    this.password = await bcrypt.hash(this.password, 12);
  }
  next();
});

export const User = model('User', userSchema);

Create a new user with User.create() or new User(data).save(), and this works exactly as intended. The password arrives hashed, every time, without the rest of your codebase needing to remember to call bcrypt.hash manually anywhere. It feels like a solved problem.

Where It Actually Breaks

// A password reset flow
export async function resetPassword(userId: string, newPassword: string) {
  await User.findByIdAndUpdate(userId, { password: newPassword });
}

This runs without error. It updates the user's password field in the database. It also never calls pre('save'), which means the password gets stored in plain text, directly in your database, with no error, no warning, and a function name, findByIdAndUpdate, that reads as a completely reasonable, idiomatic way to update a user record.

findByIdAndUpdate, findOneAndUpdate, updateOne, and updateMany all operate directly against MongoDB's update commands. They don't instantiate a full Mongoose document and call .save() on it internally, they issue an update operation straight to the database. Mongoose's document middleware, including pre('save'), only runs on document instances that actually go through .save(). These query-based update methods structurally never touch that code path at all.

Why This Specifically Gets Missed

Both approaches, .save() and findByIdAndUpdate, look like equally valid, equally idiomatic ways to update a Mongoose document, and most Mongoose tutorials use both interchangeably without flagging that they behave completely differently with respect to middleware. A codebase that uses .save() for creating users and findByIdAndUpdate for updating them, which is an extremely common and reasonable-looking pattern, ends up with password hashing that works perfectly at signup and silently doesn't exist at all for a password reset, profile update, or admin-initiated password change, unless someone happens to specifically test that exact path with a tool that shows raw database contents.

The Fix: Use Query Middleware, Not Just Document Middleware

Mongoose provides a separate middleware hook specifically for query-based update methods:

userSchema.pre('findOneAndUpdate', async function (next) {
  const update = this.getUpdate() as any;

  if (update.password) {
    update.password = await bcrypt.hash(update.password, 12);
  }

  next();
});

This specifically targets findOneAndUpdate and findByIdAndUpdate (which is implemented as a wrapper around findOneAndUpdate internally). updateOne and updateMany need their own separate middleware registration, since Mongoose treats each query method as its own distinct middleware hook rather than one unified "any kind of update" event:

userSchema.pre('updateOne', async function (next) {
  const update = this.getUpdate() as any;
  if (update.password) {
    update.password = await bcrypt.hash(update.password, 12);
  }
  next();
});

A More Defensive Alternative: Standardize on One Update Path

Registering middleware separately for every query method Mongoose offers is viable but easy to let drift out of sync as new update call sites get added later by someone who doesn't know this distinction exists. A more durable fix for anything genuinely sensitive, like password handling, is routing all updates through a single service function that always uses .save() internally, rather than letting findByIdAndUpdate calls for that model spread throughout the codebase:

// services/userService.ts
export async function updateUser(userId: string, data: Partial<UserInput>) {
  const user = await User.findById(userId);
  if (!user) throw new Error('User not found');

  Object.assign(user, data);
  await user.save(); // always goes through pre('save') middleware
  return user;
}

This trades a small amount of performance, an extra read before the write, for a guarantee that every update for this model goes through the exact same validation and transformation logic, with no separate middleware registration required for each query method and no risk of a future findByIdAndUpdate call quietly skipping it.

The Rule

Anything inside a pre('save') hook only runs for updates that go through .save(), never for findOneAndUpdate, findByIdAndUpdate, updateOne, or updateMany. For any logic where skipping it has real consequences, password hashing being the clearest example, either register the equivalent query middleware for every update method your codebase actually uses, or standardize on a single service function that always goes through .save().

I route every sensitive field update through a dedicated service layer exactly like this in the auth code behind my Next.js and MongoDB templates, specifically because this gap is easy to introduce the moment a second developer, or a future version of yourself, reaches for the more convenient-looking findByIdAndUpdate call.

Get the templates: https://pixelanas.gumroad.com

Go check your own password reset, profile update, and admin edit flows for a findByIdAndUpdate or updateOne call on a model with password hashing in pre('save'). If you find one, check what's actually sitting in that field in your database right now. Drop what you find in the comments.

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

🧠 Coder Legion Dev Log — Building the MyZubster Metaverse from the infrastructure up

Thu, 08 Oct 2026 02:32:06 +0000

🧠 Coder Legion Dev Log — Building the MyZubster Metaverse from the infrastructure up

I’ve been working on the MyZubster Metaverse with a very specific engineering approach:

do not start from the 3D layer. Start from identity, authorization, state, lifecycle and failure handling.

The current stack already includes several production-oriented building blocks.

Identity and session model

The Metaverse supports both:

guest characters
authenticated, account-linked identities

For authenticated users, canonical character data is resolved server-side so the client cannot freely override trusted identity state.

This matters because virtual-world logic becomes much harder to secure once identity, permissions and session state are mixed together.

Realtime authorization

One of the more important backend changes was introducing a fail-closed authorization path for community realtime channels.

If the membership authority is unavailable, access is denied instead of waiting on buffered database operations or leaving socket authorization in an ambiguous state.

I added coverage for:

disconnected MongoDB state
active persisted membership
membership query failures
unauthorized access attempts

This is a small change in code, but a meaningful system-design decision.

Room lifecycle

The room layer already supports:

public / authenticated / private access policies
invitations
session creation
participant join/leave
host controls
stage requests
speaker approval/revocation
participant moderation
message reporting
bounded moderation history

The important point is that the room is treated as a stateful system, not just a visual container.

Frontend state hardening

I’ve also been cleaning up React lifecycle behavior.

Recent fixes include:

reloading room state when authentication changes
stabilizing async data loading with useCallback
tightening effect dependencies
keeping UI state aligned with server-side session state

Example recent commits:

0b5e7fa — refresh Metaverse room when auth state changes
fa94a88 — stabilize frontend loading callback
0347827 — make paid analytics explicit instead of automatic

Auth and observability testing

I also aligned tests around:

authenticated Metaverse flows
server-backed sessions
auth expiry
same-origin authenticated requests
health endpoint behavior
storage failure semantics

A previous checkpoint also hardened realtime membership authorization and added test coverage around failure states:

0af298e — fail-closed realtime auth
9ac9f53 — auth session test alignment
5bf365d — auth + observability test alignment

Deployment discipline

Another part of the work was CI/CD hygiene.

I removed automatic activation of paid Vercel Web Analytics from the deployment workflow.

That means deployment stays deterministic and billing-related features remain explicit operational decisions.

Current architecture direction

The development sequence I’m following is:

identity
→ authorization
→ state
→ room lifecycle
→ moderation
→ observability
→ realtime rendering

This is intentional.

A visually impressive world with weak session handling, unclear authorization or fragile failure modes is still a fragile system.

So the current priority is to make the platform behavior coherent before connecting the final immersive realtime rendering layer.

Skills this work is exercising

This project has been a practical exercise in:

React hooks and state synchronization
Node.js backend architecture
REST API design
realtime authorization
authentication and session handling
MongoDB / Mongoose
fail-closed security patterns
moderation systems
automated testing
observability
CI/CD
Vercel deployment workflows
production debugging
incremental system hardening

The immersive layer is still the next milestone.

But the system underneath it is becoming much more real.

Build the world after the rules of the world are reliable.

CoderLegion #SoftwareEngineering #React #NodeJS #MongoDB #Realtime #SystemDesign #DevOps #WebDevelopment #Metaverse #OpenSource

Your Zod Schema's .optional() Might Be Silently Rejecting Valid Data From MongoDB

Sat, 03 Oct 2026 21:14:32 +0000

This one trips people up specifically because the two words feel interchangeable in everyday English, "optional" and "nullable" both sound like "this might not be there," and Zod treats them as two genuinely distinct, non-overlapping conditions.

What Each One Actually Validates

import { z } from 'zod';

const Schema = z.object({
  bio: z.string().optional(), // allows the key to be MISSING, or a string. Does NOT allow null.
  avatar: z.string().nullable(), // requires the key to be PRESENT, as a string OR null. Rejects missing.
});

.optional() accepts undefined, meaning the field can be genuinely absent from the object entirely, but if it's present, it must match the base type, never null. .nullable() accepts null specifically, but the key itself still needs to actually be present in the object, just potentially holding the value null rather than a real value. These aren't two flavors of the same leniency, they're validating two different, non-overlapping conditions, and passing the wrong kind of absence to either one fails validation.

Where This Actually Breaks With MongoDB Specifically

MongoDB documents very commonly store an explicit null for a field that has no value, rather than omitting the key entirely, especially for fields that were part of the original schema design but just haven't been set for a given document yet.

// A real document from MongoDB
{
  _id: '...',
  name: 'Alex',
  bio: null, // explicitly null, not missing
}

// ❌ This schema expects the key to be absent, not present-and-null
const UserSchema = z.object({
  name: z.string(),
  bio: z.string().optional(),
});

UserSchema.safeParse(userFromDb); // fails: bio received null, optional() doesn't accept null

bio: null is a completely valid, common shape coming straight out of MongoDB, and .optional() rejects it anyway, since .optional() only ever expected undefined, never null. The validation failure here isn't catching a real data problem, it's a mismatch between what Zod's .optional() actually means and what the developer assumed it meant when writing the schema against real database output.

The Fix: Match the Schema to What the Data Actually Does

const UserSchema = z.object({
  name: z.string(),
  bio: z.string().nullable(), // correctly matches MongoDB's explicit null
});

For a field that MongoDB stores as explicit null when empty, .nullable() is the correct validator, not .optional(). If a field could genuinely be either missing entirely or explicitly null, both conditions need to be allowed together:

bio: z.string().nullable().optional(), // accepts a string, null, OR a missing key

Chaining both is completely valid and often the most accurate reflection of real-world data that's passed through several code paths, some of which omit the key, others of which explicitly set it to null.

Why This Doesn't Show Up Until Real Data Hits It

A schema tested only against hand-written test objects during development, where the developer naturally omits fields they don't care about rather than explicitly setting them to null, never actually exercises this gap. The validation failure only shows up once real documents, containing MongoDB's genuine, common pattern of explicit null values, flow through the same schema, often well after the schema was written and seemingly working correctly.

A Quick Way to Check Which One You Actually Need

Query a real, representative document from your actual collection and look at the field in question directly. If it shows up as null in the raw document, you need .nullable(). If the key is genuinely absent from documents where it has no value, you need .optional(). Guessing based on the field's name or what "feels right" is exactly how this mismatch gets introduced in the first place, checking the real data removes the ambiguity entirely.

const doc = await User.findOne({}).lean();
console.log(Object.keys(doc).includes('bio'), doc.bio);
// If bio is a key with value null → nullable()
// If bio is simply absent as a key → optional()

The Actual Rule

.optional() and .nullable() are not interchangeable leniency settings, they validate two distinct, specific shapes of absence, and MongoDB's own default behavior of storing explicit null means .nullable() is very often the one you actually need for fields coming straight out of a database, not .optional(), even though "optional" is the word that intuitively comes to mind first. When a schema seems to be rejecting data that looks completely fine in your database, this mismatch is one of the first things worth checking.

I ran into exactly this building the MongoDB-backed dashboards covered in the MongoDB patterns I use in every Next.js project, and it's one of those small schema details worth getting right once rather than re-discovering on every new project.

If a Zod schema is rejecting data that looks completely valid sitting in your actual database, check whether it's this exact .optional() versus .nullable() mismatch before assuming the data itself is somehow wrong. Drop what you find in the comments.

Get the templates: https://pixelanas.gumroad.com

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

generateMetadata and Your Page Component Are Probably Querying the Same Data Twice

Sat, 19 Sep 2026 10:48:58 +0000

I wrote earlier about cache() fixing redundant database queries when multiple components in a tree independently call the same query function. There's one specific, extremely common place this exact pattern shows up that's worth calling out directly, because it's not an edge case, it's the default shape of nearly every dynamic page with SEO metadata.

The Setup Almost Every Dynamic Page Has

// app/blog/[slug]/page.tsx
import { getPost } from '@/lib/queries/posts';

export async function generateMetadata({ params }) {
  const { slug } = await params;
  const post = await getPost(slug); // query #1

  return {
    title: post.title,
    description: post.excerpt,
  };
}

export default async function BlogPost({ params }) {
  const { slug } = await params;
  const post = await getPost(slug); // query #2, same post, same request

  return (
    <article>
      <h1>{post.title}</h1>
      <div>{post.content}</div>
    </article>
  );
}

This is the completely standard pattern from the Next.js metadata setup covered in an earlier post, and it's genuinely necessary, generateMetadata needs the post's title and excerpt for the meta tags, the page component needs the post's full content to actually render it. Both run for every single request to this page. Both call getPost(slug) independently. Without a specific fix, that's two full database round trips for what is, semantically, the exact same piece of data, fetched twice, for every single page view.

Why This Specific Case Matters More Than It Might Seem

This isn't a rare pattern someone might occasionally write, it's the default shape of generateMetadata combined with any dynamic page, which means it's quietly doubling database load on nearly every content-driven page across an entire site, blog posts, product pages, any dynamically rendered detail page with SEO metadata. A site with meaningful traffic on these pages is paying for twice the database queries it actually needs, on some of its most frequently visited pages, for no functional reason.

The Fix: Wrap the Query in React's cache()

// lib/queries/posts.ts
import { cache } from 'react';
import { connectDB } from '@/lib/db';
import Post from '@/models/Post';

export const getPost = cache(async (slug: string) => {
  await connectDB();
  return Post.findOne({ slug }).lean();
});

That's the entire fix, no changes needed anywhere else. generateMetadata and the page component both still call getPost(slug) exactly as before, the code looks identical. What changes is that React's request memoization now recognizes both calls, same function, same argument, within the same render pass, and only actually executes the underlying query once, returning the cached result for the second call.

Why This Is Worth Checking Specifically, Not Just Generally

The earlier post on this topic covered the general pattern, multiple components in a tree calling the same query. This specific case is worth checking on its own because it's so easy to miss precisely because generateMetadata and the page component don't look like they're part of the same "component tree" in the way nested components obviously are. They're two separate exported functions in the same file, and it's genuinely easy to write both, each looking completely correct in isolation, without ever noticing they're duplicating a database call for the same request.

A Quick Way to Verify This Is Actually Happening

Add a temporary log inside the un-cached query function and load the page once:

export async function getPost(slug: string) {
  console.log('getPost called for:', slug); // temporary
  await connectDB();
  return Post.findOne({ slug }).lean();
}

If you see that log line twice for a single page load, that's this exact issue, confirmed directly rather than assumed. After wrapping the function in cache(), the same test should show it logging only once per request.

Where This Doesn't Apply

If generateMetadata and the page component genuinely need different data, metadata pulling from a lighter, separate summary endpoint while the page fetches full content from somewhere else entirely, there's no duplication to fix, since they're not actually calling the same underlying function with the same arguments in the first place. This fix specifically applies when both are calling the identical query function, which, for most blog and content-driven page setups, is exactly the common case.

The Actual Rule

Any query function called both inside generateMetadata and inside the page component it belongs to should be wrapped in cache(), as close to a universal default as a rule gets in this specific area. It costs nothing to add, changes no calling code, and directly eliminates a real, silent, doubled database query that exists by default on essentially every dynamically rendered page with SEO metadata.

Go check any page in your own project using generateMetadata alongside a matching page component, specifically whether the underlying query function is wrapped in cache(). If it's not, you're very likely double-querying on every single page load for that route. Drop what you find in the comments.

Get the templates: https://pixelanas.gumroad.com

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

How I Eliminated 19,494 Duplicate MongoDB Records (And What I'd Do Differently Today)

Sat, 19 Sep 2026 08:33:15 +0000

At a previous organization, I was handed ownership of a job platform serving 1,000+ users worldwide. Jobs on the platform could be scoped to a single city, a state, a whole country, or "available worldwide." Simple enough. At least, you'd think so.

When I joined, this was early in my career and I was a frontend dev. I was interested in backend work but hadn't done any real backend engineering yet — I wasn't battle-tested. Somewhere along the line, I got handed the whole platform. Frontend, backend, all of it. Bracing moment. I forced myself to slow down, actually understand the system end to end, and build a real plan before touching anything.

After talking with product management, I pulled up the backend task backlog. I learn best by doing, so I started at the bottom — smallest bugs first, working my way up. It went well for a while. Then I hit a ticket tied to core business logic.

The Wake-Up Call

Some endpoints were clocking API response times north of 7,000ms. I genuinely thought it was a typo. Then I remembered running into something similar on the frontend side before — so I patched the UX so users never felt the full weight of the delay. But the actual problem was still sitting there, untouched.

Before touching any code, I made a checklist. We were running MongoDB with NestJS:

What schemas was the endpoint touching?
What optimization had already been applied to those schemas?
How was the endpoint fetching data?
How many records did each related schema currently hold?

Checklist in hand, I went into the codebase.

My innocent dev brain was not prepared for the chaos I found.

What Was Actually Happening

Here's the setup: assume an empty jobs collection. A recruiter creates a job listing scoped to New York City. The backend takes the payload and creates one record with location set to New York City. Straightforward.

Now assume another empty collection, and this time the job is scoped to the United States instead of a single city. What do you think happens?

If you guessed "the same thing — one record, location set to United States" — I have bad news for you.

What the backend actually did: it looked at the payload, recognized "United States" as a country rather than a city, looked up every known city in the U.S. — and created a separate job record for each one.

Read that again.

Instead of one record, the platform generated one identical record per city, differing only in _id and location. And it got worse: a job marked "available worldwide" would spawn a record for every city on the planet.

This was a live product with real traffic. Querying that collection — even with indexes in place — was a nightmare.

For context on scale: a single job scoped to just the USA generated 19,495 records — one canonical listing plus 19,494 exact duplicates.


Database records (1 USA-wide job)	19,495
Exact duplicates among them	19,494
Query time on affected endpoints	~2.3s (worst cases exceeded 7s for worldwide-scoped jobs)

My first reaction was horrified confusion. After talking it over with management, I learned there was a reason for the original design — I won't get into it here, it's not the point of this piece. What mattered was that I had free rein to fix it if I could pull it off. Hold my Pokémon card.

The Constraints

Fixing the architecture wasn't the hard part conceptually — the hard part was fixing it under real constraints:

We still needed to resolve which cities a job was available in, for every existing feature that depended on that.
Whatever I built had to slot into the existing infrastructure without a rewrite of everything downstream.
I needed to show measurable, provable performance gains — not just "trust me, it's better."
None of this could disrupt daily active users. No maintenance windows, no downtime.

Designing the Fix

I started with the schema. The core question: how do I preserve full location fidelity while collapsing tens of thousands of records into one?

I thought back to basic compression concepts — take data, strip it to a minimal representation, and pair it with the instructions needed to reconstruct the original. That's essentially what I needed here: a compact way to represent "this job is available in X" without one document per X.

Two deliverables:

A new, efficient schema.
A migration path that collapsed the old duplicated records into the new schema.

For the new schema, I kept every existing field and replaced the flat location string with a structured locationObj:

interface LocationObj {
  country: string | string[];
  state: string | string[];
  city: string | string[];
}

The logic:

Single city → city is a string, state and country resolve to that city's parent state/country.
A specific group of cities → city becomes a string array.
An entire state → city becomes the sentinel value "all", state holds the actual state name.
An entire country → state and city both become "all".
Worldwide → country, state, and city all become "all".

So a job available across Berlin, Munich, and Hamburg looks like:

{
  title: "Marketing Manager",
  company: "TechCorp",
  locationObj: {
    country: "Germany",
    state: "all",
    city: ["Berlin", "Munich", "Hamburg"]
  }
}

And a job available anywhere in the world:

{
  title: "Software Developer",
  company: "TechCorp",
  locationObj: {
    country: "all",
    state: "all",
    city: "all"
  }
}

One document. Full location semantics preserved. No fan-out.

Migrating Without Downtime

The deduplication piece ran as a batch script against the old collection: group records that shared everything except location (same title, company, description, salary — the actual "identity" of a job), reconstruct the correct locationObj from the set of location values in that group, and write a single new document into the new collection.

Once the new collection was validated against the old one, I built new endpoints on top of the new schema, mirroring the contracts of the old ones, and prepared the frontend to point at them.

The cutover itself was a backfill with two catch-up passes, not a one-shot migration:

Run the migration script once against the live old collection.
Right before deploying the new endpoints, re-run it — this catches any jobs created while the first pass was running, since the old collection kept accepting writes the entire time.
Deploy the new endpoints and switch the frontend over.
Re-run the script once more right after — this catches anything created under the old schema during the deploy window itself, in case the rollout wasn't instantaneous.

I tested the full path — frontend, backend, and DB — on local and staging before any of this touched production, so by the time it shipped, the swap itself was just a normal deploy. From a user's perspective, they kept seeing exactly the jobs they were supposed to see, the whole way through.

Querying the New Schema

The query logic has to walk the hierarchy — country, then state, then city — because "all" at any level short-circuits the rest:

const query = {
  $or: [
    { "locationObj.country": "all" },
    { "locationObj.country": userCountry, "locationObj.state": "all" },
    { "locationObj.country": userCountry, "locationObj.state": userState, "locationObj.city": "all" },
    { "locationObj.country": userCountry, "locationObj.city": userCity },
    { "locationObj.country": userCountry, "locationObj.city": { $in: [userCity] } }
  ]
};

Combined with a compound index on locationObj.country, locationObj.state, and locationObj.city, this scales linearly with the number of actual jobs — not the number of cities they happen to be available in.

The Results

Metric	Before	After	Change
Database records (1 USA-wide job)	19,495	1	99.99% fewer
Query time	~2.3s	~0.12s	~95% faster
Storage footprint	847 MB	0.043 MB	99.99% smaller
Index size	156 MB	0.008 MB	99.99% smaller
Memory usage	~4.2 GB	~0.21 MB	99.99% less RAM
Exact duplicate documents	19,494	0	Eliminated

Management was, unsurprisingly, impressed.

How I'd Do It Now

Looking back at this with a few more years under my belt, the schema still holds up. What I'd change is the migration and verification process — at the time, "backfill, then re-run twice around the cutover" felt rigorous. It worked. But it also relied on timing more than I'd want to admit if I were reviewing this today.

The more rigid version of this — the one I'd actually reach for now — is the expand-contract pattern:

Expand: Deploy dual-writes behind a feature flag — every job creation or update writes to both the old and new schema at the same time.
Sync: Use MongoDB Change Streams to mirror any old-schema writes into the new schema in real time during the transition, instead of relying on timed reruns of a migration script. This closes the race-condition window entirely — there's no gap to miss, because nothing is waiting for a scheduled pass.
Contract: Once reads are fully on the new schema and stable, kill the old write path.

I'd also make the migration script itself idempotent and checkpointed — track the last processed _id so a crash mid-run means it resumes instead of reprocessing or silently skipping documents. Mine was a script I babysat through completion; a checkpointed version is one I could walk away from.

On verification — "compare the counts" was good enough to convince management, but it wasn't rigorous. I'd now do a real reconciliation pass: diff the reconstructed locationObj against the actual set of cities in the old schema, per job, and flag any mismatch automatically instead of trusting a spot check. And before flipping reads over, I'd run a dark-launch period — querying both schemas in production and logging discrepancies without serving the shadow result to users — so any gap shows up before it's user-facing, not after.

One more thing I'd change that isn't about the migration at all: I'd want this caught before it ever shipped. The root failure was that nothing stopped a single job-creation event from silently producing thousands of documents. A write-layer guardrail — "creating one job should never result in more than one document, alert if it does" — plus a five-line regression test asserting exactly that, would have caught this in review instead of in production months later.

Lessons Learned

Embedding vs. referencing isn't the real question — normalization is. The original design wasn't wrong because it embedded or referenced incorrectly; it was wrong because it materialized every possible resolution of a hierarchical value instead of storing the hierarchy itself. The general MongoDB advice of "embed for read-heavy, reference for write-heavy" didn't apply here — the fix was closer to a compression problem than a modeling problem.

Sentinel values are underrated. Using "all" as a stand-in for "every value at this level" let one field type (string | string[]) represent five distinct scopes without a single boolean flag or separate scope enum.

Get the actual scale of the problem before designing the fix. I didn't just need "make it faster" — I needed a number to prove it. Knowing it was 19,495 records for one job is what made the case to management, and it's what makes this story worth telling.

Fields, numbers, and code above are reconstructed from a project where I owned the full backend under an NDA — details are generalized where necessary, but the architecture, the failure mode, and the metrics are real.

Cover photo by Mika Baumeister on Unsplash

Building MyZubster: A Circular Marketplace Connected to a Social Metaverse

Mon, 14 Sep 2026 12:43:38 +0000

Building MyZubster: A Circular Marketplace Connected to a Social Metaverse
MyZubster is an open-source ecosystem that connects a circular marketplace, local communities, educational support, underground culture, visual storytelling and an experimental social metaverse.
The project is based on a cyclical model:
Discover → Connect → Exchange → Use → Learn → Repair → Reuse → Recover → Contribute
Instead of building another isolated marketplace or virtual world, MyZubster connects these activities through shared navigation, account-linked characters and measurable user journeys.
The MyZubster Marketplace
The Marketplace is the operational discovery layer of the ecosystem. It organizes products, services, skills and community activities into accessible categories and subcategories.
The current demo explores areas including:

local food and independent producers;
kefir and fermented products;
underground music and alternative culture;
sound-system equipment;
event and creative services;
university and educational assistance;
programming and digital services;
repair and reuse;
recycling and material recovery;
sustainable everyday products.
Explore the Marketplace:
🛒 https://www.myzubster.com/marketplace?utm_source=coderlegion&utm_medium=article&utm_campaign=myzubster_circular_funnel&utm_content=marketplace_overview
Why Kefir Is Part of the Ecosystem
Kefir is a useful example of how a small local product can become part of a larger circular process.
A visitor can discover a producer, learn about the production process, contact the seller and understand what happens to the packaging after consumption.
Local production
↓
Marketplace discovery
↓
Informed purchase
↓
Consumption
↓
Container reuse
↓
Recycling or material recovery
MyZubster does not automatically certify food safety, production standards or health benefits. Those claims require evidence and compliance from the responsible producer.
The platform’s role is to make sellers, information and provenance easier to discover and inspect.
Open the local-products Marketplace funnel:
🌱 https://www.myzubster.com/marketplace?category=food&utm_source=coderlegion&utm_medium=article&utm_campaign=myzubster_circular_funnel&utm_content=kefir_local_producers
Supporting Underground and Independent Culture
Underground communities often depend on informal networks connecting musicians, artists, technicians, venues and equipment owners.
MyZubster can provide optional infrastructure for discovering:
DJs, musicians and independent performers;
sound-system equipment;
audio repair specialists;
rehearsal and event services;
visual artists and graphic designers;
alternative clothing and records;
independent venues and cultural initiatives;
reusable creative materials.
A sound system can move through a complete circular lifecycle:
Equipment discovery
↓
Purchase or rental
↓
Event use
↓
Maintenance
↓
Repair
↓
Resale and reuse
↓
Component recovery
The objective is not to control underground culture. It is to give independent communities tools they can use on their own terms.
Explore the underground and sound-system funnel:
🔊 https://www.myzubster.com/marketplace?category=sound-system&utm_source=coderlegion&utm_medium=article&utm_campaign=myzubster_circular_funnel&utm_content=underground_sound_system
University and Educational Assistance
Education is another important part of the MyZubster cycle.
Students can discover legitimate support such as:
tutoring;
programming guidance;
language practice;
research-method explanations;
document and presentation formatting;
portfolio support;
study groups;
guidance toward internships;
open-source contribution opportunities.
Educational support should help students learn. It must not be used to impersonate students, complete prohibited examinations or submit deceptive academic work.
The educational cycle is:
Ask for support
↓
Learn
↓
Practise
↓
Build a project
↓
Document the work
↓
Contribute
↓
Help another learner
Explore the educational Marketplace funnel:
🎓 https://www.myzubster.com/marketplace?category=education&utm_source=coderlegion&utm_medium=article&utm_campaign=myzubster_circular_funnel&utm_content=university_help
Entering the MyZubster Metaverse
The MyZubster Metaverse is the visual and social navigation layer of the ecosystem.
Its current environment, Neon Plaza, allows visitors to:
enter as guests or authenticated users;
use account-linked characters;
move around a shared environment;
see other active visitors;
communicate through chat and emotes;
discover nearby participants;
visit ecosystem portals;
complete an exploration mission;
preserve visited-zone progress;
discover experimental virtual rooms.
Enter Neon Plaza:
🪐 https://www.myzubster.com/metaverse?utm_source=coderlegion&utm_medium=article&utm_campaign=myzubster_circular_funnel&utm_content=metaverse_entry
Account-Linked Characters
A visitor can explore without creating a verified identity claim.
Authenticated users can use persistent account-linked characters. The server determines which character belongs to the authenticated account instead of trusting a name supplied by the browser.
The current system includes:
canonical server-side character profiles;
account-linked character ownership;
authenticated profile retrieval;
server-backed mission progress;
visited-zone persistence;
live player presence;
session recovery;
expired-JWT handling;
protected character-progress updates.
A display name is not considered legal identity evidence. The MYZ VERIFIED label indicates that the character is linked to a MyZubster account, not that a person’s legal identity, professional skills or qualifications have been certified.
Mission Progress
Visitors receive an initial mission: move through Neon Plaza and reach one of its portals.
For guests, progress remains local to the browser. For authenticated characters, visited zones can be stored on the account-linked server profile.
The server accepts only known landmark identifiers. It derives character ownership from the verified authentication token and does not trust a character name submitted by the client.
This creates a safer progression model:
Authenticated account
↓
Canonical character
↓
Visit a valid landmark
↓
Server validates the landmark
↓
Progress is attached to the account-owned character
Try the mission funnel:
🧭 https://www.myzubster.com/metaverse?utm_source=coderlegion&utm_medium=article&utm_campaign=myzubster_circular_funnel&utm_content=first_mission
Experimental Virtual Rooms
MyZubster is beginning to connect Neon Plaza with persistent virtual rooms.
The discovery system applies access rules on the server:
guests can discover public rooms;
authenticated users can also discover account-only rooms;
private rooms remain hidden;
drafts and archived rooms are excluded;
only published, scheduled or live spaces are discoverable.
These rooms are still experimental. A dedicated room experience will be introduced only after navigation, access control, moderation and lifecycle handling are ready.
Follow the virtual-room funnel:
🏛️ https://www.myzubster.com/metaverse?utm_source=coderlegion&utm_medium=article&utm_campaign=myzubster_circular_funnel&utm_content=virtual_rooms
MyZubster Visual
MyZubster Visual provides the storytelling layer.
Illustrations, interactive popups and visual narratives can explain connections between:
physical products;
circular lifecycles;
environmental projects;
characters;
community contributions;
Marketplace categories;
Metaverse destinations.
A popup can introduce a story, provide additional context and direct a visitor toward the next relevant part of the ecosystem.
The intended funnel is:
Visual story
↓
Interactive popup
↓
Context and explanation
↓
Marketplace category or project
↓
Metaverse exploration
↓
Community participation
Explore the visual-story funnel:
🎨 https://www.myzubster.com/fumetto?utm_source=coderlegion&utm_medium=article&utm_campaign=myzubster_circular_funnel&utm_content=visual_popups
How the Complete Circular Funnel Works
Consider a visitor interested in locally produced kefir:
1. They discover MyZubster through this article.
2. They open the Marketplace.
3. They explore local food and producer listings.
4. They learn about the product and its provenance.
5. They contact or purchase from an authorized seller.
6. They reuse or correctly recycle the packaging.
7. They enter the Metaverse.
8. They visit the Marketplace or LIFE Projects portal.
9. They document or share a useful contribution.
10. That contribution helps another visitor discover the circular pathway.
  The same model can apply to underground sound equipment:
11. Discover equipment.
12. Use it for an event.
13. Maintain and repair it.
14. Resell or reuse it.
15. Recover components at the end of its usable life.
16. Share repair knowledge with the community.
  It can also apply to educational support:
17. Request tutoring.
18. Learn a skill.
19. Build a project.
20. Publish the code.
21. Contribute to MyZubster.
22. Help another learner.
  Technical Architecture
  The current implementation combines:
React for the user interface;
Node.js and Express for backend services;
MongoDB for persistent characters, progress and room records;
authenticated API routes;
server-authoritative access rules;
short-lived presence and chat records;
GitHub for open-source development;
Vercel for deployment and web analytics.
The architecture maintains an important boundary:
The client presents the experience, but the server controls identity, ownership, persistence and access decisions.
Measuring the Funnel
Each link in this article contains UTM parameters designed to distinguish traffic from Coder Legion.
The campaign uses:
utm_source=coderlegion
utm_medium=article
utm_campaign=myzubster_circular_funnel
The utm_content value identifies the selected journey:
marketplace_overview
kefir_local_producers
underground_sound_system
university_help
metaverse_entry
first_mission
virtual_rooms
visual_popups
github_repository
This makes it possible to compare which concepts attract attention and which destinations visitors choose next.
Build With Us
MyZubster is being developed openly. Developers can inspect the implementation, review the architecture, propose improvements and contribute code.
💻 https://github.com/MyZubster-Ecosystem/myzubster?utm_source=coderlegion&utm_medium=article&utm_campaign=myzubster_circular_funnel&utm_content=github_repository
The project remains an evolving prototype. Experimental functionality, verified evidence and operational services are kept clearly distinguishable.
The objective is not simply to create another marketplace or virtual environment. It is to build a connected system in which software helps communities discover opportunities, learn, exchange value, extend product lifecycles and create new contributions.
Explore the Ecosystem
🛒 Marketplace Demo
https://www.myzubster.com/marketplace?utm_source=coderlegion&utm_medium=article&utm_campaign=myzubster_circular_funnel&utm_content=marketplace_overview
🪐 MyZubster Metaverse
https://www.myzubster.com/metaverse?utm_source=coderlegion&utm_medium=article&utm_campaign=myzubster_circular_funnel&utm_content=metaverse_entry
🎨 MyZubster Visual
https://www.myzubster.com/fumetto?utm_source=coderlegion&utm_medium=article&utm_campaign=myzubster_circular_funnel&utm_content=visual_popups
💻 Open-source Repository
https://github.com/MyZubster-Ecosystem/myzubster?utm_source=coderlegion&utm_medium=article&utm_campaign=myzubster_circular_funnel&utm_content=github_repository
MyZubster #OpenSource #WebDevelopment #Metaverse #Marketplace #CircularEconomy #ReactJS #NodeJS #MongoDB #BuildInPublic

That .lean() Query Optimization Might Be Leaking Password Hashes in Your API Response

Fri, 11 Sep 2026 08:29:25 +0000

I've recommended .lean() in earlier posts, it's genuinely the right call for read-heavy queries, faster, lighter, no Mongoose document overhead. There's a real gap it opens up that's worth calling out specifically, because it silently undoes a common, otherwise-good security pattern, and the failure is invisible unless you specifically go check the actual API response.

The Setup That Looks Like a Solid Security Pattern

// models/User.ts
const UserSchema = new Schema({
  name: String,
  email: String,
  password: { type: String, select: false }, // excluded from queries by default
}, {
  toJSON: {
    transform: (doc, ret) => {
      delete ret.password; // extra safety net, strips it even if somehow selected
      delete ret.__v;
      return ret;
    },
  },
});

This is a reasonable, defense-in-depth setup, select: false keeps the password out of normal queries by default, and the toJSON transform is a second safety net stripping it from anything that does get serialized, in case it was ever explicitly selected somewhere. Two layers, looks solid.

Where This Quietly Breaks

// app/api/users/route.ts
export async function GET() {
  const users = await User.find().select('+password').lean();
  // some legitimate internal reason to select password here, comparing hashes, an admin tool, etc.
  return Response.json({ users });
}

.lean() returns plain JavaScript objects, not real Mongoose documents. That toJSON transform you configured on the schema is a method that exists on Mongoose document instances specifically, it never runs on a plain object. Response.json({ users }) calls JSON.stringify under the hood, which on a lean plain object just serializes every property directly, no transform, no stripping, nothing. If password was selected anywhere in that query chain, it goes straight into the API response, completely bypassing the safety net you specifically built to prevent exactly that.

Why This Is Genuinely Easy to Introduce

The select('+password') and .lean() combination often gets added at different times, for different reasons, by different people, or by the same developer months apart with the original toJSON safety net long forgotten. Someone adds .lean() to a query for a legitimate performance reason, weeks or months after the select('+password') was added for some specific internal need, comparing a password during a migration script, an admin debugging tool, and neither change alone looks dangerous in isolation. The combination is what's dangerous, and nothing about reviewing either change independently reveals that.

The Actual Fix: Never Rely on toJSON as Your Only Safety Net

The real lesson here isn't "don't use .lean()", it's genuinely great for performance and you should keep using it. The lesson is that toJSON transforms are not a reliable safety net for anything you might combine with .lean(), so the actual stripping needs to happen explicitly, regardless of whether the query happens to be lean or not.

// lib/serializers/user.ts
export function toSafeUser(user: any) {
  const { password, __v, ...safe } = user;
  return safe;
}

// app/api/users/route.ts
import { toSafeUser } from '@/lib/serializers/user';

export async function GET() {
  const users = await User.find().lean(); // no need to select password here at all
  return Response.json({ users: users.map(toSafeUser) });
}

An explicit serializer function works identically whether the input came from a lean query or a real document, since it's just plain object destructuring, not relying on a Mongoose-specific lifecycle hook that only fires under certain conditions.

The Broader Rule This Points To

Never assume a security-relevant transform tied to a document lifecycle method still applies once you've introduced .lean(), .toObject(), or any pattern that returns a plain object instead of a real document instance. toJSON, toObject transforms, and virtuals are all Mongoose document features, tied specifically to the document class, not properties of the raw data itself. The moment you're working with a plain object, none of that schema-level behavior comes along automatically, and any security assumption riding on it needs to be re-verified explicitly, not assumed to still hold.

Check Your Own API Routes

grep -rn "lean()" --include="*.ts" app/api/

For each match, check two things, does the underlying query ever select a genuinely sensitive field, and does the response actually get filtered explicitly, not just relying on a schema-level transform that a lean query never triggers. If a route selects a sensitive field for some legitimate internal reason and also calls .lean(), that combination is worth verifying directly against the actual JSON response, not just the code.

If you've got .lean() anywhere near a query that ever selects a genuinely sensitive field, go check the actual raw response, not just the code, this specific combination is exactly the kind of thing that looks completely fine on read-through and only reveals itself in the real serialized output. Drop what you find in the comments.

Get the templates: https://pixelanas.gumroad.com

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

🚀 Building a Production Facebook Messenger → AI Pipeline with MyZubster & Zorgax

Sun, 06 Sep 2026 14:45:08 +0000

🚀 Building a Production Facebook Messenger → AI Pipeline with MyZubster & Zorgax

We just shipped a new integration in the MyZubster open-source ecosystem: Zorgax, our AI copilot, can now receive and answer messages directly through Facebook Messenger.

The production flow is:

Messenger → Meta Webhook → MyZubster API → Zorgax → Meta Send API → Messenger

Getting the webhook to return 200 was only part of the job.

One interesting production issue appeared after the webhook was already receiving Messenger events correctly: Meta received its 200 OK, but the asynchronous Zorgax processing that followed could be interrupted in the serverless environment.

The solution was to keep Meta's acknowledgement fast while explicitly keeping the background work alive with Vercel's waitUntil().

We also implemented and verified:

Meta webhook challenge verification
X-Hub-Signature-256 request validation
secure environment-based App Secret and Page Access Token handling
Messenger event processing
MongoDB connectivity
Zorgax AI processing
Meta Send API responses
multilingual conversations
production deployment on Vercel

The final production trace now looks like:

POST /api/meta/messenger/webhook → 200
Connected to MongoDB
[meta-messenger] {"event":"message_handled","mode":"zorgax"}

And, most importantly, the response actually makes the full trip back to the Messenger conversation. 🎯

The code is open source

The serverless/background-processing fix was shipped in:

PR #1007 — fix(messenger): keep Zorgax reply alive after webhook ack

Read PR #1007 on GitHub

Explore the MyZubster repository

Visit MyZubster

This was a good reminder that “the webhook returned 200” doesn't necessarily mean the workflow completed—especially when asynchronous AI work happens after the HTTP response in a serverless environment.

Now Zorgax has another gateway into the MyZubster ecosystem, and the implementation is available for other developers to inspect, improve, and build on.

Open source. Production tested. Community built.

OpenSource #NodeJS #AI #FacebookMessenger #MetaDevelopers #Vercel #Webhooks #APIs #MongoDB #BuildInPublic #MyZubster #Zorgax

Next.js Automatically Dedupes fetch() Calls. Your MongoDB Queries Don't Get That for Free.

Fri, 04 Sep 2026 10:14:16 +0000

Here's something that trips up almost everyone moving from a fetch-based API to a direct MongoDB connection in Next.js, and it's costing real database load on projects that don't even realize it's happening.

Next.js automatically deduplicates fetch() calls with identical parameters within a single render pass. If your layout, your page, and three nested components all call fetch('/api/user'), Next.js is smart enough to only actually make that request once and reuse the result everywhere else it's called. This is called request memoization, and it's one of the quieter, genuinely useful things the App Router does for you automatically.

Direct database queries get none of this. Nothing.

What This Actually Looks Like

// lib/queries/user.ts
import { connectDB } from '@/lib/db';
import User from '@/models/User';

export async function getCurrentUser(userId: string) {
  await connectDB();
  return User.findById(userId).lean();
}

// app/dashboard/layout.tsx
export default async function DashboardLayout({ children }) {
  const user = await getCurrentUser(userId); // query #1
  return <div><Sidebar user={user} />{children}</div>;
}

// app/dashboard/page.tsx
export default async function DashboardPage() {
  const user = await getCurrentUser(userId); // query #2, same user, same request
  return <Header user={user} />;
}

// components/UserMenu.tsx (Server Component, rendered inside the page)
export default async function UserMenu() {
  const user = await getCurrentUser(userId); // query #3
  return <Menu user={user} />;
}

Three separate calls to getCurrentUser, same user ID, same single incoming request from the browser. If this were fetch('/api/user') instead of a direct Mongoose query, Next.js would automatically collapse these into one actual network call. Since it's a direct database query, none of that automatic deduplication applies. This is three real round trips to MongoDB, for data that was identical all three times, on every single page load.

Why This Doesn't Show Up in Development

On a local database with almost no latency and no real load, three redundant queries feel instant, indistinguishable from one. This is exactly why it's so easy to ship without noticing. The cost only becomes visible under real production conditions, real network latency to your database, real concurrent traffic, where those extra round trips add up into measurably slower page loads and meaningfully higher database load than the page actually needed.

The Actual Fix: React's cache() Function

React ships a cache() function specifically for this. It memoizes a function's result within a single render pass, exactly the same behavior fetch gets automatically, just applied manually to whatever function you wrap with it.

// lib/queries/user.ts
import { cache } from 'react';
import { connectDB } from '@/lib/db';
import User from '@/models/User';

export const getCurrentUser = cache(async (userId: string) => {
  await connectDB();
  return User.findById(userId).lean();
});

That's the entire fix. Every call to getCurrentUser with the same userId, anywhere in that same render pass, layout, page, nested component, now returns the same cached promise instead of triggering a new query. Three calls become one actual database hit.

The Part People Get Wrong About cache()

This is request-scoped, not persistent across requests. It deduplicates calls within one single render of one single incoming request, then resets completely for the next request. It is not the same thing as unstable_cache, which persists data across multiple requests and needs explicit revalidation.

// cache() - deduplicates within ONE request, resets every time
export const getCurrentUser = cache(async (userId: string) => { ... });

// unstable_cache - persists ACROSS requests, needs revalidateTag to update
export const getPosts = unstable_cache(async () => { ... }, ['posts'], {
  revalidate: 3600,
});

Mixing these up causes two different problems. Using cache() where you needed unstable_cache means you're still hitting the database on every single request, just once instead of three times. Using unstable_cache where you needed cache() means you're persisting data across requests when you actually just wanted to avoid redundant calls within one page load, which can serve stale data longer than intended.

Where I'd Actually Add This

Not everywhere, that would be over-engineering a solution for calls that only ever happen once per request anyway. The pattern worth watching for specifically: any query function, getCurrentUser, getTenant, getSession-adjacent lookups, that reasonably gets called from multiple places in the same component tree, a layout, a page, and a few nested components all independently needing "who is the current user."

Check Your Own Project

Grep your query functions for ones called from more than one component in the same tree, and check whether they're wrapped in cache(). If they're not, and they're not already going through unstable_cache or an equivalent persistent layer, you likely have the exact redundant-query pattern above running silently in production right now.

Curious how many people actually knew about this before reading this post versus assumed Next.js handled it automatically the same way it does for fetch. Drop your honest answer in the comments, genuinely curious how well-known this actually is.

Get the templates: https://pixelanas.gumroad.com

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

Building MyZubster in Public: Marketplace, ZORGAX, LIFE Pilot, and a Verified Metaverse

Mon, 31 Aug 2026 00:33:06 +0000

MyZubster has recently completed one of its most important development phases so far.

We worked across the entire stack: production infrastructure, authentication, Marketplace visibility, economic APIs, digital entrepreneurship workflows, security automation, and public Metaverse identities.

This is a technical overview of what we built, what broke along the way, and what is now running in production.

The MyZubster ecosystem

MyZubster is an open-source ecosystem that combines several connected services:

A community Marketplace
GitHub-based user identities
ZORGAX, an assistant and economic experimentation layer
A guided digital entrepreneurship program called LIFE Pilot
A persistent Metaverse called Neon Plaza
Public contributor and project records
Automated testing, security, and deployment workflows

The project is available on GitHub:

github.com/MyZubster-Ecosystem/myzubster

The production platform is available at:

www.myzubster.com

Fixing the production foundation

Before adding new functionality, we had to solve several production-level problems.

The application was deployed, but some important features were not working reliably.

The main issues included:

MongoDB operations timing out
GitHub OAuth failing after authorization
Marketplace data not loading in production
Backend requests being captured by frontend routes
Node.js runtime incompatibility with the MongoDB driver
Dependency lockfile differences during clean Vercel builds

MongoDB and GitHub OAuth

GitHub authentication could start correctly, but the login process failed when the application attempted to query the users collection.

The operation remained buffered until it timed out.

We corrected the production MongoDB configuration and verified that the application could connect to the database before executing authentication queries.

We also aligned the production runtime with Node.js 20, which provides a stable environment for the current MongoDB driver and application dependencies.

After the fix:

GitHub OAuth completes successfully
User records can be retrieved or created
Marketplace listings can access persistent data
API routes no longer fail because of buffered database operations

Making the Marketplace visible

The Marketplace was already present in the repository, but it was not correctly exposed through the production application.

We restored its route, connected it to the live API, and verified that the listing endpoint returned a successful response from production.

The Marketplace is now accessible here:

Open the MyZubster Marketplace

This provides the base for future services, contributor offers, digital products, and community exchanges.

ZORGAX pricing and access model

ZORGAX now exposes a public pricing API with three plans:

Zorgax Free
Zorgax Pro
Zorgax Developer

The current Pro reference price is €9.90 per month.

The Developer plan is intended for users who need:

API access
Automation
Higher usage limits
Advanced workspace functionality

The payment architecture is explicitly non-custodial.

MyZubster does not request or accept users’ private keys, and it does not automatically sign blockchain transactions.

External settlement must be independently verified before paid access is activated.

This separation is critical:

A payment request is not proof of payment, and an unconfirmed transaction is not confirmed revenue.

Building the ZORGAX economic layer

ZORGAX has evolved from a basic assistant into a more structured economic experimentation system.

We added services and APIs for:

Confirmed economic inflows
Capital-allocation proposals
Advisory capital recommendations
Capital decision tracking
Outcome recording
Decision memory
Bounded learning

Confirmed inflows only

The economic layer does not treat every payment intent as available capital.

Only independently confirmed inflows are included in the capital model.

This prevents pending, rejected, or unverified transactions from influencing recommendations.

Advisory capital allocation

ZORGAX can generate proposals about how available resources could be allocated.

These recommendations are advisory. They do not automatically transfer money or execute financial operations.

A server-side policy constrains how the recommendations are generated.

Decision and outcome memory

Capital decisions can be recorded together with their eventual outcomes.

The system can use this history to improve later recommendations through bounded learning.

Bounded learning means the system can learn from evidence while remaining inside explicit operational limits.

It cannot silently expand its own authority.

From idea to digital product

We also built a structured digital entrepreneurship workflow.

The workflow can guide a project through the following stages:

Project creation
Idea intake
Idea validation
Product blueprint generation
Offer creation
Launch preparation
Metrics collection
Evidence-based learning
Product iteration

Each stage has dedicated backend services, API routes, data models, and automated tests.

Digital idea validation

A user can submit a digital product idea and receive a structured validation result.

The validation engine evaluates the idea and creates a persistent snapshot.

This makes it possible to compare different iterations instead of overwriting previous reasoning.

The workflow connects validation results to the project so that later stages can use real evidence.

Product blueprint engine

A validated idea can be converted into a digital product blueprint.

The blueprint provides a structured foundation for the product, including its direction, audience, and execution model.

The latest blueprint is persisted with the project and exposed through a dedicated API.

This creates a bridge between an idea and an actionable product plan.

Launch and offer engine

The launch-and-offer engine converts the product blueprint into a controlled offer package.

Instead of stopping at strategy, the workflow prepares something that can be presented to real users and tested.

The resulting package can then be connected to product metrics and learning events.

Metrics and learning

A digital product needs more than a launch plan. It needs measurable feedback.

We added:

Product metric event models
Metric collection services
Public API routes
Learning summaries
Tests for metric aggregation and learning behavior

The objective is to create a loop:

Idea
  ↓
Validation
  ↓
Blueprint
  ↓
Offer
  ↓
Launch
  ↓
Metrics
  ↓
Learning
  ↓
Next iteration
The public response does not expose:
- Email addresses
- Password data
- Password hashes
- Internal MongoDB identifiers
- Private GitHub identifiers
- Private verification metadata
- Authentication credentials
The database query also uses an explicit field projection so that unnecessary data is not loaded before the response is created.
Security automation
Security work was added directly to the development workflow.
The repository now includes:
- Automated security checks
- Continuous evidence gates
- Pull-request security verification
- Post-remediation security records
- Public security documentation
- Regression protection for critical routes
The latest Marketplace remediation recorded zero remaining HIGH-severity findings.
The goal is to make security evidence reproducible.
Instead of saying that a feature is secure, the project records what was tested and protects the result with automated checks.
Test coverage
Tests were added across the new services and integration boundaries.
The coverage includes:
- Marketplace behavior and security
- Metaverse public statistics
- Verified explorer responses
- ZORGAX pricing and product APIs
- Capital recommendations
- Capital decision lifecycle
- Confirmed inflow aggregation
- Digital idea validation
- Product blueprint generation
- Launch offer generation
- Metrics and learning
- Pilot enrollment
- Pilot onboarding
- First-project creation
- Idea ranking
- LIFE Pilot API routing
- Vercel route precedence
The final H4x0r release passed backend route tests and frontend syntax validation before deployment.
Production verification
The latest changes were committed to the main branch and deployed automatically through Vercel.
After deployment, we verified:
- The deployment reached the Ready state
- The production API returned HTTP 200
- H4x0r appeared in featuredCharacters
- Only approved public fields were present
- No sensitive fields were exposed
- The frontend displayed the verified explorer card
- The public GitHub link worked correctly
The release commit is available here:
feat(metaverse): publish verified explorer profiles
Engineering lessons
This development phase produced several useful lessons.
1. Production configuration is part of the codebase
A correct application can still fail because of runtime versions, environment variables, dependency metadata, or proxy routing.
2. Public APIs need explicit boundaries
Never expose complete database documents by default.
A public API should intentionally select and transform every field it returns.
3. Payment detection and payment confirmation are different states
Paid access should not be activated from a payment intent alone.
Settlement must be independently verified.
4. Automation needs authority limits
An intelligent system can generate recommendations and learn from outcomes without receiving permission to execute financial actions.
5. Integration tests protect the real product
Many important failures happen between components rather than inside individual functions.
Authentication, databases, frontend routing, deployment proxies, and settlement verification all require integration-level protection.
What comes next?
The next stages of MyZubster development include:
- Completing verified paid-access activation
- Expanding Marketplace services
- Adding more verified explorers to Neon Plaza
- Improving the LIFE Pilot user experience
- Strengthening production monitoring
- Publishing clearer governance and contributor records
- Exploring resilient and decentralized service distribution
- Continuing security and privacy improvements
MyZubster is still evolving, but it now has a much stronger foundation.
The ecosystem has persistent identities, working authentication, public services, economic experiments, guided project workflows, privacy-safe APIs, automated tests, and a verified production deployment.
If you want to inspect the code or contribute, visit:
MyZubster Ecosystem on GitHub
You can explore the live platform here:
www.myzubster.com

How to Build a SaaS with Next.js and MongoDB in 2026 (Complete Guide)

Fri, 28 Aug 2026 09:41:09 +0000

Building a SaaS product from scratch involves the same handful of pieces every time. Authentication, a database that actually scales, subscription billing, and a dashboard people can use without a manual. None of these are individually hard. Getting all of them working together, correctly, is where most side projects stall out.

This is the full path, from an empty folder to something you could actually charge people for, using Next.js 15 and MongoDB.

Step 1: Set Up the Foundation

Start with the App Router, TypeScript, and a folder structure that will not need restructuring once the project grows. Route groups separate your public marketing pages, authenticated dashboard, and API routes cleanly, without affecting the actual URLs.

src/
├── app/
│   ├── (marketing)/     ← public landing pages
│   ├── (auth)/          ← login, signup
│   ├── (dashboard)/     ← authenticated app
│   └── api/webhooks/    ← Stripe, etc.
├── lib/
├── models/
└── actions/

Getting this right early avoids the painful restructuring that happens when a flat app/ folder grows past a dozen pages with no separation between public and authenticated routes.

Step 2: Connect MongoDB the Right Way

A cached connection pattern is non-negotiable for anything deployed on Vercel or another serverless platform. Without it, hot reloads in development and cold starts in production both create new connections faster than MongoDB wants to accept them.

// lib/db.ts
import mongoose from 'mongoose';

let cached = (global as any).mongoose || { conn: null, promise: null };

export async function connectDB() {
  if (cached.conn) return cached.conn;
  if (!cached.promise) {
    cached.promise = mongoose.connect(process.env.MONGODB_URI as string);
  }
  cached.conn = await cached.promise;
  return cached.conn;
}

From here, every model, User, Subscription, and whatever your product's core data actually is, follows the same pattern, checking mongoose.models.X before defining a new one so hot reload does not throw an overwrite error.

Step 3: Authentication

JWT stored in an httpOnly cookie keeps the token out of reach of client-side JavaScript, which closes off the most common token-theft path through XSS. Middleware checks the cookie before a protected route even starts rendering, and a second check inside the dashboard layout catches anything middleware might miss.

// middleware.ts
export function middleware(request: NextRequest) {
  const session = request.cookies.get('auth-token');
  if (!session && request.nextUrl.pathname.startsWith('/dashboard')) {
    return NextResponse.redirect(new URL('/login', request.url));
  }
}

Role-based access, separating "not logged in" from "logged in but not allowed here", sits on top of this once you have more than one type of user, admin versus regular user, for example.

Step 4: Subscription Billing with Stripe

This is the step people try to shortcut, and it always causes problems later. Do not update subscription status based on the success redirect after checkout. Browsers close, redirects fail to load, and your database never finds out what actually happened.

Webhooks are the only reliable source of truth:

// app/api/webhooks/stripe/route.ts
case 'checkout.session.completed': {
  const session = event.data.object as Stripe.Checkout.Session;
  await User.findByIdAndUpdate(session.metadata?.userId, {
    subscriptionStatus: 'active',
  });
  break;
}

The rest of your app reads subscriptionStatus straight from the database, never calling Stripe directly just to check if a user has access. The webhook is what keeps that field accurate.

Step 5: The Dashboard Itself

Server Components handle almost all of the actual data fetching, no client-side loading spinners for data that is available the moment the page renders. Search, filters, and pagination live in the URL rather than component state, which keeps a filtered view shareable and refresh-safe.

// app/(dashboard)/customers/page.tsx
export default async function CustomersPage({ searchParams }) {
  const { search, page } = await searchParams;
  const { customers, totalPages } = await getCustomers({ search, page });

  return (
    <div>
      <CustomerFilters />
      <CustomerList customers={customers} />
      <Pagination currentPage={Number(page) || 1} totalPages={totalPages} />
    </div>
  );
}

Client Components only wrap the pieces that are actually interactive, a filter input, a modal, a button with local state, keeping the JavaScript shipped to the browser as small as it can reasonably be.

Step 6: Forms, Validation, and Error Handling

One Zod schema per form, shared between client-side validation (instant feedback as someone types) and the Server Action itself (the real check, since client validation can always be bypassed). Route-level error.tsx files catch anything unexpected without taking down the rest of the app, and notFound() handles missing records cleanly as an actual 404 rather than a crash.

Step 7: Email and Notifications

Verification emails, password resets, and payment receipts are expected in any real SaaS. Triggering the receipt email from inside the Stripe webhook, not from the client-side redirect, keeps it reliable for the same reason the subscription status update needs to happen there. Writing templates as actual React components, rather than raw HTML strings, makes this part of the stack far less painful to maintain.

Step 8: Real-Time Features, If You Need Them

Not every SaaS needs this, but anything with a live status, a queue, a dashboard that multiple people watch at once, benefits from pushing updates instead of polling. Trigger events from the server after a database write, and treat the database as the source of truth, with real-time events as a fast notification layer on top rather than a replacement for it.

Step 9: Rate Limiting and Security

Public endpoints, a contact form, a signup route, need rate limiting before they see real traffic, not after something abuses them. A sliding window limiter keyed by IP for unauthenticated routes, or by user ID for authenticated ones, catches both accidental retry loops and deliberate abuse. Every route still needs its own input validation on top of this, since rate limiting controls how often, not what gets sent.

Step 10: SEO for the Marketing Pages

The dashboard itself does not need to rank in search results, but the marketing pages around it do. Unique metadata per page through generateMetadata, a generated sitemap that updates automatically as content is added, and canonical URLs to avoid a www versus non-www split are the baseline. From there, actual content, a blog, comparison pages, documentation, is what builds the organic traffic a SaaS needs to reduce dependence on paid acquisition.

What This Actually Takes to Build From Scratch

Realistically, a solo developer building all of this correctly, connection caching, auth, billing, a working dashboard, emails, basic security, is looking at real weeks of work before the first paying customer could even sign up, longer if any of these pieces are unfamiliar.

This is exactly the gap a well-built template closes. Not because the concepts are hard to learn, everything above is a pattern, not a secret, but because assembling all of them correctly, with the pieces actually wired together instead of built in isolation, is where the time actually goes.

Frequently Asked Questions

Is MongoDB a good choice for a SaaS product, or should I use a SQL database?
MongoDB works well for SaaS products with flexible or evolving data shapes, since schema changes do not require a migration in the same way a relational database does. For data with strict relational integrity requirements, financial ledgers, complex multi-table joins, a SQL database like Postgres is often a better fit. Many SaaS products end up using both, MongoDB for flexible application data, a relational database for anything requiring strict consistency.

How long does it take to build a SaaS MVP with Next.js?
For an experienced developer working solo, a focused MVP covering auth, one core feature, and basic billing typically takes several weeks of consistent work. Scope is the biggest variable, an MVP that tries to include every feature from the start almost always takes longer than one built around a single, well-defined core workflow.

Do I need a separate backend, or can Next.js handle the whole SaaS?
For most SaaS products, Next.js alone is sufficient. Server Actions and API routes handle backend logic, and Server Components handle data fetching, without needing a separate Express or Node server running alongside it. A separate backend becomes worth considering only for very specific needs, like a long-running background process that does not fit a serverless request-response model.

If you want to see this exact architecture already built, rather than assembling it piece by piece, I sell Next.js and MongoDB SaaS templates with auth, billing, and a working dashboard already wired together.

Browse templates: https://pixelanas.gumroad.com

Building something specific and want a hand with it? Get in touch: https://pixelanas.com/contact

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

How I Used PostgreSQL Events and MongoDB Event Logs in Two Different Real-World Systems

Sun, 05 Jul 2026 09:20:40 +0000

I built two systems that both needed event tracking.
I used PostgreSQL in one and MongoDB in the other.
Here's exactly why — and what I learned from doing both.
Although the two systems were completely different in purpose, they shared one architectural requirement: each needed a reliable way to record everything that happened — not just the final state, but the story of how that state came to be.
That sounds simple, but the nature of the events in each system was so different that using the same database for both would have caused real problems. This article explains how I approached event tracking in each system, why the solutions diverged, and what the experience taught me about choosing the right persistence model.

PostgreSQL Events and Projections in the Delivery Backend

The delivery backend was built around structured workflows: vendors create orders, couriers pick them up, jobs move through well-defined states, and batches group multiple deliveries. Every action in this system has a clear relational meaning. An event like ORDER_CREATED or JOB_STARTED isn't just a log entry — it's part of the domain model.
Because of that, PostgreSQL was the natural home for events. Each event needed to be tied to a specific entity, validated against constraints, indexed for fast lookup, and projected into timeline tables that the dashboard could query efficiently.
The event log table looked like this:

model EventLog {
  id          Int             @id @default(autoincrement())
  entityType  EventEntityType
  entityId    Int
  eventType   String
  payload     Json
  createdAt   DateTime        @default(now())

  @@index([entityType, entityId, createdAt])
}

Every event is immutable. Every event belongs to a relational entity. Every event carries structured metadata.
Once an event is written, it is immediately transformed into a timeline entry:

switch (event.entityType) {
  case EventEntityType.ORDER:
    await this.orderTimelineProjection.project(event);
    break;

  case EventEntityType.JOB:
    await this.jobTimelineProjection.project(event);
    break;
}

This projection layer is what makes the system feel fast. Instead of running complex joins to reconstruct history, the dashboard reads from dedicated timeline tables optimized for sequential queries. It also makes debugging easier: if something goes wrong, I can replay the event stream and see exactly how the system reached its current state.
PostgreSQL excels in this environment because the events are structured, relational, and part of the domain's consistency model. They aren't just analytics — they're part of the truth.

MongoDB Event Logs and Aggregation in Waitless

Waitless is a completely different kind of system. It's a real-time queue management platform where customers join via QR code, track their position live, and leave without creating an account. The system generates a lot of events — but these events are not relational. They're behavioral.
A CUSTOMER_JOINED event might contain the customer's position.
A NO_SHOW event might contain a timestamp.
A CUSTOMER_LEFT event might contain a reason.
A CALLED_NEXT event might contain which token was called.

The shape of the metadata changes depending on the event type. Forcing this into a rigid SQL schema would either require dozens of nullable columns or a JSON column that PostgreSQL would struggle to index efficiently at scale.
MongoDB is built for exactly this. Each event is a document with flexible metadata:

@Schema({ timestamps: true })
export class QueueEvent {
  @Prop({ required: true })
  queueId!: string;

  @Prop({ required: true, enum: QueueEventType })
  eventType!: QueueEventType;

  @Prop()
  token?: string;

  @Prop({ type: Object })
  metadata?: Record<string, any>;
}

This flexibility becomes powerful when combined with MongoDB's aggregation pipeline. Waitless computes analytics like total joined, total arrived, total no-shows, completion rate, and drop-off rate — all derived from event counts, not relational queries.
A note on Redis: Waitless already had Redis in the stack backing BullMQ. But Redis is a key-value store optimized for job queues and caching — not for querying across documents or running aggregations. MongoDB was the right fit for this specific need.
The separation between MongoDB and PostgreSQL is also crucial for performance. If analytics were computed in PostgreSQL, every dashboard refresh would compete with real-time queue operations for the same connection pool.

If I had used PostgreSQL for Waitless events, every analytics query would hit the same connection pool as the queue state — meaning a slow dashboard query could delay a customer joining a queue.
MongoDB avoids this entirely. It handles flexible event shapes, high write throughput, and analytics workloads without interfering with the transactional state stored in PostgreSQL.

Why PostgreSQL for One and MongoDB for the Other?

The answer becomes clear once you see the nature of the data.
In the delivery backend, events are part of the domain model. They describe state transitions, belong to relational entities, and require strong consistency. PostgreSQL is the right tool.
In Waitless, events are part of analytics. They are flexible, metadata-heavy, and high-volume. They have no relational meaning — they're behavioral observations about what happened in a queue session. MongoDB is the right tool.
Using PostgreSQL for both would have created unnecessary coupling and performance bottlenecks. Using MongoDB for both would have sacrificed relational guarantees where they mattered most.
This is the essence of polyglot persistence: choose the database that matches the shape and purpose of the data, not the one you're most comfortable with.

The Question That Actually Matters

Building two event-driven systems taught me that "SQL vs NoSQL" is the wrong question entirely.
The real question is: what role do your events play in the system?
If events belong to the domain model — use a relational database.
If events belong to analytics — use a document database.
PostgreSQL gave me structured timelines and strong consistency. MongoDB gave me flexible event logs and fast aggregation. Both systems use events, but the events serve completely different masters.
Understanding that difference before writing a single line of code is what separates a working system from a well-designed one.

MongoDB DR Drill Automation with Terraform, Python & Jenkins — How We Made Restores Boring

Thu, 02 Jul 2026 15:09:45 +0000

Backups Don't Save You. Restores Do.

We ran a MongoDB restore drill last quarter. It failed — not the restore itself, but the confidence. Nobody in the room was sure the data was actually intact. The service came back up, and we all just stared at each other.

That was the problem. So we fixed it by automating everything.

One Jenkins job now provisions infra, builds the replica set, restores from dumps, validates data integrity, and stores a full audit trail. Here's exactly how it works.

The Goal

Remove every manual, error-prone step from the DR process:

Identical restore flow across all environments
Automated replica set setup — no manual rs.initiate() typos
Real validation that proves data is intact, not just assumed
Full audit trail for post-mortems and compliance reviews

The Pipeline: 5 Stages

1. Infrastructure with Terraform

Every drill starts with clean infra. Terraform provisions EC2s, networking, and persistent volumes from scratch — same starting point every time. No leftover state. No "works on my machine" surprises.

resource "aws_instance" "mongo_node" {
  count         = 3
  ami           = var.mongo_ami
  instance_type = "t3.medium"
  tags = {
    Name = "mongo-dr-node-${count.index}"
    Role = "mongodb-replica"
  }
}

2. Replica Set Creation (Python)

Instead of manually running rs.initiate() and rs.add() and hoping the timing works, a Python script handles the entire setup — ordering, retries, and confirmation.

from pymongo import MongoClient
import time

def init_replica_set(primary_host, secondary_hosts):
    client = MongoClient(f"mongodb://{primary_host}:27017")
    config = {
        "_id": "rs0",
        "members": [{"_id": i, "host": h}
                    for i, h in enumerate([primary_host] + secondary_hosts)]
    }
    client.admin.command("replSetInitiate", config)
    # Wait for PRIMARY election
    for _ in range(30):
        status = client.admin.command("replSetGetStatus")
        if any(m["stateStr"] == "PRIMARY" for m in status["members"]):
            return True
        time.sleep(2)
    raise Exception("Replica set did not elect a PRIMARY in time")

Automating this removes timing issues and misconfiguration. Every replica set comes up the same way.

3. Backup & Restore

Backups are normalized into compressed archives. The restore unpacks a dump and applies it to the fresh nodes:

# Create dump
mongodump --host $SOURCE_HOST --db $DB_NAME \
  --out /backup/dump --gzip

# Restore to DR environment
mongorestore --host $DR_HOST --db $DB_NAME \
  /backup/dump/$DB_NAME --gzip --drop

4. Validation & Comparison — The Part Most Teams Skip

This is the step that actually builds confidence. The validation script:

Checks which collections exist (flags missing collections)
Compares document counts collection by collection
Compares indexes between source and restored DB
Samples _id values for obvious data mismatches

def validate_restore(source_uri, dr_uri, db_name):
    src = MongoClient(source_uri)[db_name]
    dr  = MongoClient(dr_uri)[db_name]

    report = {"status": "pass", "collections": {}}

    for col in src.list_collection_names():
        src_count = src[col].count_documents({})
        dr_count  = dr[col].count_documents({})
        src_idx   = sorted(src[col].index_information().keys())
        dr_idx    = sorted(dr[col].index_information().keys())

        match = (src_count == dr_count) and (src_idx == dr_idx)
        report["collections"][col] = {
            "count_match":  match,
            "source_count": src_count,
            "dr_count":     dr_count,
            "index_match":  src_idx == dr_idx
        }
        if not match:
            report["status"] = "fail"

    return report

Exit code 0 = counts and indexes match → Jenkins passes.
Non-zero = mismatch → Jenkins fails the build immediately.

No more guessing. No more staring at each other in the war room.

5. Jenkins Orchestration

Single Jenkins pipeline. Stages run sequentially, each one gated on the previous:

pipeline {
  agent any
  stages {
    stage('Provision Infra') {
      steps {
        sh 'terraform init && terraform apply -auto-approve'
      }
    }
    stage('Setup Replica Set') {
      steps {
        sh 'python3 scripts/init_replica_set.py'
      }
    }
    stage('Restore MongoDB') {
      steps {
        sh 'bash scripts/restore.sh'
      }
    }
    stage('Validate Restore') {
      steps {
        sh 'python3 scripts/validate_restore.py'
      }
    }
    stage('Archive Logs') {
      steps {
        archiveArtifacts artifacts: 'reports/*.json, logs/*.log'
      }
    }
  }
}

Every run is logged, every report is archived. When auditors ask if restores work — you show them a report with timestamps, counts, and index diffs. Not a gut feeling.

Lessons Learned

Automate infra, not just the restore. Terraform gives you a clean slate every drill. Manual infra setup introduces variability that hides real problems.

Validation is not optional. A restore that "seems fine" is not the same as a restore that is fine. Document count mismatches and missing indexes are easy to catch automatically and impossible to catch by eyeballing logs.

Logs equal trust. The audit trail is what makes your DR process credible to others — engineers, management, auditors. Without it, you're asking people to take your word for it.

Minimal input reduces errors. We trimmed required inputs to just host + DB name and let scripts infer the rest. Less to type = fewer mistakes under pressure.

Practice makes permanent. Each drill found a small improvement. After ten drills, the process was genuinely fast and boring — which is exactly what you want.

The Outcome

We went from a 3-hour manual war room exercise to a single Jenkins job anyone can trigger. The drills are now predictable, repeatable, and quick.

More importantly — everyone on the team believes the restores work, because the validation script proves it every single time.

Boring DR is good DR.

Running MongoDB in production? When did you last drill a full restore? Drop your setup in the comments — curious how teams handle validation.

🏗️ CasePilot — India's AI Legal Case Worker

Mon, 08 Jun 2026 12:16:33 +0000

Most Indians facing landlord fraud, consumer scams, or workplace violations have no idea where to start. Legal fees cost more than the dispute is worth. CasePilot changes that.

What it does:
Describe your legal situation in plain language → 7 AI agents investigate your case → get a ready-to-send legal notice in minutes, for free.

The stack:
→ Google ADK — 7-agent pipeline (Intake → Law Research → Evidence → Strategy → Citation → Drafting → Memory)
→ MongoDB Atlas Vector Search — 3,310 chunks across 13 Indian laws, semantic search at 3,072 dims
→ MongoDB MCP Server — agents query Atlas via MCP protocol
→ Gemini 3.1 Flash-Lite — reasoning across all agents
→ Google Cloud Run — live public API
→ gemini-embedding-2 — document + law corpus embeddings

What makes it different from legal chatbots:
❌ Not: "Here's what Section 108 says"
✅ Yes: "Your landlord violated TPA S.108. Here's a pre-filled legal notice. Send it today. File consumer complaint in 15 days if no response."

Evidence uploaded → Justice Score calculated → action timeline built → case persists across sessions.

Live demo:

https://casepilot-34309631370.us-central1.run.app

Built solo for the Google Cloud Rapid Agent Hackathon — MongoDB track.

GitHub: github.com/ssurekumar01111-hue/casepilot

From CRUD APIs to Production-Style Backend Architecture: My Video Sharing Platform Project

Fri, 29 May 2026 09:30:58 +0000

Completed a Production-Style Video Sharing Platform Backend API
Over the past few weeks, I've been building a backend system inspired by modern video-sharing platforms using Node.js, Express.js, MongoDB, and Cloudinary.

This project went far beyond CRUD operations and helped me understand how real backend systems are designed.

Features Implemented:
✅ JWT Authentication & Refresh Token Rotation
✅ Secure HTTP-Only Cookie Based Sessions
✅ Video Upload & Cloudinary Integration
✅ Comment System
✅ Like System (Videos, Comments, Tweets)
✅ Subscription System
✅ Playlist Management
✅ Watch History Tracking
✅ Creator Dashboard Analytics
✅ Pagination & Aggregation Pipelines

Key Things I Learned:
Authentication using Access & Refresh Tokens
Designing relationships in MongoDB
MongoDB Aggregation Pipelines ($lookup, $group, $project, $addFields)
Ownership-Based Authorization
Cloudinary Media Management
Cascade Cleanup Logic
Building consistent API response structures

Interesting Debugging Challenges:
Duplicate MongoDB index issues
Populate path errors
Refresh token validation flow
Route mismatches
Aggregation pipeline bugs
Watch history synchronization
Video deletion cleanup across collections

Documentation Created:
Complete API Documentation
Authentication Flow Documentation
️ Database Schema Documentation

What's Next?
Building the React frontend and converting this into a complete full-stack application.

GitHub Repository: https://github.com/SairajJadhav08/Video-Sharing-Platform-API

Tooling Tips: Visualizing Your Data in MongoDB and Kafka

Wed, 06 May 2026 00:32:14 +0000

Hey folks!

Quick tooling post today. Before moving forward to the next services, I want to share the tools I use to visualize and inspect data in MongoDB and Kafka during development.

Without these tools, debugging an asynchronous data pipeline would be much harder — you're essentially "blind" to what's happening inside the containers.

MongoDB — MongoDB Compass

MongoDB Compass is MongoDB's official GUI. Free and available for Windows, Mac and Linux.

What you can do:

Browse collections and documents
Run queries with visual filters
See indexes created by Flyway/code
Inspect document schemas
Monitor performance metrics

How to connect to the local container:

URI: mongodb://root:password@localhost:27017/?authSource=admin

After connecting, you'll see the market_data_db database with the price_history collection being populated by broker-market-data-api on each ingestion round.

Download: mongodb.com/products/tools/compass

Kafka — Offset Explorer (formerly Kafka Tool)

Offset Explorer is a GUI for inspecting Kafka clusters. It lets you see topics, partitions, messages and offsets without needing the command line.

What you can do:

List all topics in the cluster
See messages from each partition in real time
Inspect the content (key + value) of each message
Monitor consumer group offsets
Check whether consumers have lag

How to connect to the local container:

Bootstrap Servers: localhost:9092

After connecting, you'll see the trading-assets-market-data-v1 topic with messages published by broker-market-data-api. Each message will have the ticker as key (e.g. PETR4) and the quote JSON as value.

Download: kafkatool.com

RabbitMQ — Management UI (already included)

RabbitMQ comes with a built-in web interface. In our docker-compose.yml, port 15672 is exposed:

URL: http://localhost:15672
User: admin
Pass: admin_pass

What you can do:

View queues, exchanges and bindings
Inspect messages in queues
Manually publish messages to test consumers
Monitor message rates and lag
See the DLQ (Dead Letter Queue) when messages fail

This is the most useful tool for debugging the flow between trading-broker-order and b3-matching-engine.

Workflow Tip

During development, I use all three tools in parallel:

Offset Explorer to confirm broker-market-data-api is publishing messages to Kafka
MongoDB Compass to confirm the history is being persisted
RabbitMQ Management to confirm orders are arriving in the B3 queues

This turns an opaque asynchronous pipeline into something completely observable.

What's Next?

With the tools set up, in the next post we build b3-market-sync — the Java service that consumes quotes from Kafka and synchronizes them to Redis, feeding the Matching Engine with real-time prices.

About the Series

⬅️ Previous Post: Market Data Integrator

➡️ Next Post: Syncing the Real Market: Consuming Brapi and Feeding Redis with Spring Boot

Series Index: Series Roadmap

Links:

Show & Tell: Building a Self-Healing SEO Architecture for a Vue SPA

Fri, 01 May 2026 18:26:49 +0000

Hey CoderLegion!

I recently tackled a massive SEO and scaling hurdle for my app, Recipe Finder, and wanted to share the architecture breakdown with the community. If you're building SPAs and relying on external APIs, you've probably run into this exact nightmare.

Building a modern Single Page Application (SPA) with Vite and Vue is great for user experience, but it's a minefield for SEO. We faced three major hurdles:

Aggressive Bot Protection: Our .htaccess was so tight it was blocking crawlers that we actually wanted.

The "SPA Meta Trap": Social media bots (Facebook, WhatsApp) couldn't read our dynamic recipe titles or images because they don't execute JavaScript.

The Scale Problem: We have access to millions of recipes via the Spoonacular API, but we don't own the full database. How do you tell Google about millions of pages you don't physically store?

️ The Tech Stack
Frontend: Vite + Vue 3 (Hosted on Apache)

Backend: Node.js + Express (Hosted on Firebase Functions)

Database: MongoDB

Provider: Spoonacular API

The Solution: A 3-Step "Self-Healing" Architecture

Solving the Social Preview (The Meta Injection)
Since our frontend is on a standard Apache host, we couldn't use edge functions easily. Instead, we optimized our URL structure to include SEO-friendly slugs:

recipe-finder.org/recipe/644488-german-rhubarb-cake-with-meringue

We then implemented a backend-driven meta-injection strategy. When a recipe is requested, our Express server pre-fills the Open Graph tags (og:title, og:image, og:description) using the recipe summary, ensuring beautiful previews on social media.

The "Self-Building" Database
We didn't want to scrape millions of recipes (and get banned). Instead, we created an Organic Growth Engine.

Every time a user (guest or authenticated) clicks a recipe, our Express backend performs an Upsert into MongoDB. If it's a new recipe, it's added to our "SEO Index." If it's an existing one, we update the lastViewed timestamp.

JavaScript
// Remove stale entry, then push back to front with a fresh timestamp
await recipeViewedModel.findOneAndUpdate(
  { auth0Id },
  { $pull: { recipes: { id: recipe.id } } }
);

await recipeViewedModel.findOneAndUpdate(
  { auth0Id },
  { $push: { recipes: { $each: [{ ...recipe, viewedAt: new Date() }], $position: 0 } } },
  { upsert: true, new: true }
);

This ensures our database only grows with high-quality, relevant content that users actually care about.

The Dynamic Hybrid Sitemap
A static sitemap.xml was impossible for millions of potential links. We built a Dynamic Sitemap Index:

sitemap-main.xml — A static file on our hosting server for core pages (Home, Tools, About).

sitemap-recipes-[n].xml — Dynamic routes on Express that query MongoDB and generate XML on the fly in 50,000-unit chunks.

The Master Index — A central sitemap.xml that bridges the two, served via a silent proxy in .htaccess.

Apache
RewriteRule ^sitemap\.xml$ https://your-region-your-project.cloudfunctions.net/api/sitemap.xml [R=301,L]
RewriteRule ^sitemap-recipes-([0-9]+)\.xml$ https://your-region-your-project.cloudfunctions.net/api/sitemap-recipes-$1.xml [R=301,L]

Any request for a sitemap is silently routed to the Express API, which assembles the XML from MongoDB on the fly — no static file maintenance required.

The Results
Google Search Console Verified: Live URL testing shows Google successfully rendering the SPA and reading the dynamic content.

Automated SEO: The more our users cook, the larger our sitemap grows. We don't have to manually add a single link.

Zero-Maintenance Scaling: The system handles 10 recipes or 10 million with the same memory footprint thanks to MongoDB's $group and $limit aggregations.

Key Takeaway for CoderLegion
Don't build for millions of pages on Day 1. Build a system that lets your users' activity grow your SEO footprint for you. Work with the bots, not against them.

Author: Rusu Ionut

Project: recipe-finder.org

Discussion: How is everyone else handling SEO for large-scale SP

Created a new feature on Recipe-Finder.org, FREE Month of Premium(https://recipe-finder.org/invite-)

Wed, 29 Apr 2026 06:35:59 +0000

We recently built the referral page for Recipe Finder: https://recipe-finder.org/invite-one-free-month. The goal was to make referrals feel like a real product feature, not just a static landing page.

On the frontend, we used

Vue 3
TypeScript
Vite
Vuetify

to build the page and referral UI.
For auth we used Auth0, and on the backend we used Node.js + Express, deployed with Firebase Functions, with MongoDB/Mongoose handling referral codes, invite tracking, and reward state.
The flow is simple: a user shares their referral link, the app captures the code from the URL, a new user claims it securely, and once Stripe confirms a paid Premium signup, we automatically extend the inviter’s Premium by 30 days.

Give it a try: https://recipe-finder.org/invite-one-free-month

From localStorage to Database: Understanding the MongoDB Structure

Wed, 19 Nov 2025 08:09:32 +0000

Intro

Hey everyone! If you are a developer who is just starting to work with databases, or perhaps you already know the basics and are looking for a recap or a more in-deph understanding, I hope this article will help you.
When I learn something new, I like to dig deeper and understand the underlying logic to ensure everything is clear. That's why I have structured this article starting from the NoSQL fundamentals all the way to the internal data storage mechanisms, concentrating on the why behind the database. So, I wish you happy reading!

In this article, we will study:

#1. Where we Started: The SQL World

When we work with Relational Database like MySQL or PostgreSQL, our data lines up neatly in those perfectly organized tables. We have to make sure every single row follows a rigid column blueprint, decided right from the start. If we ever need to switch up that structure, we basically hit the brakes, tackle complex changes, and cross our fingers that all the existing data still fits.
We got consistency and logic out of this, sure, but as our apps kept scaling, that stiffnes became a real bottleneck for moving fast.

#2. The NoSQL story: Not Only SQL
When it comes to NoSQL databases, you might have heard that it means No SQL, but let's clear things up: it's not a total war against SQL! The correct and modern interpretation is Not Only SQL❗

Why 'Not Only SQL'❓

Because we relize that SQL is a fantastic certain tasks (where strict relationship and consistency are vital), today's modern applications (social networks, gaming, websites) demand speed and flexibility. NoSQL is a supplement, an alternative that allows us to choose the right tool for the right job. We aren't giving up on SQL, we're just adding more powerful to our toolkit.

#3. From localStorage to Database
Among the many types of NoSQL databases, MongoDB falls in to the Document-Oryented category.

Imagne no longer storing data across separate rows and columns. Instead, you store a complete record, for example, a user profile with all their addresses, orders, and preferences, in a single package called a Document.

Document:

→ The Document is the basic building block. It looks exactly like a JSON (JavaScript Object Notation) object, but, technically speacking, MongoDB uses BSON (Binary JSON). Why BSON❓ Because it's a faster binary format that also includes data types that simple JSON dosen't have (like date/ time values).

→ All these Documents (which, note, do not have to look identical) are grouped into a Collection, a kind of folder that replaces the old tables. To get an idea, we can think of documents as the value stored in localStorage, but they are much more powerful than a simple string value in localStorage❗

→ The Document is the smallest and most important unit of data in MongoDB, being the complete package of information that offers tremendous flexibility by allowing:

→ Storing nested objects,
→ Storing lists,
→ Direct storage of complex data types (such as Date/Time).

Here we have a Document:


{    // unique ID 
  "_id": ObjectId("61a5c6f0e8f0b7c4d5a9b8c7"),
     // complex data type: Date/Time
  "order_date": ISODate("2025-11-18T10:00:00Z"), 
  "status": "Shipped",
      // nested object
  "shipping_address": {
    "street": "10 Example Blvd",
    "city": "London",
    "zip_code": "SW1A 0AA"
  },
    // list (Array)
  "items": [
    {
      "product_name": "Laptop Pro",
      "quantity": 1,
      "price": 1200.00
    },
    {
      "product_name": "Mouse Wireless",
      "quantity": 2,
      "price": 25.50
    }
  ]
}

Colection:

→ If a Document is the equivalent of an individual record (like a complex value stored in localStorage or a row in a SQL table), the Collection in MongoDB is the folder that holds all those records together.

→The Collection is the conceptual equivalent of a Table in a relational (SQL) database. Unlike a SQL table, where all rows must adhere to the same structure (the same columns), Documents within a MongoDB Collection can have different structures (dynamic schema)❗

→ A small example, In MongoDB, we have a Database named school. This Database contains two Collections: students and teachers.
,

→ 1. The students Collection contains two documents:

  {
    first_name: "Alice",
    last_name: "Smith",
    age: 16,
    grade: 10,
    enrollment_date: new Date("2024-09-01T00:00:00Z"),
    courses: ["Math", "Physics", "English"] // Array
  },

  {
    first_name: "Bob",
    last_name: "Johnson",
    age: 17,
    grade: 11,
    enrollment_date: new Date("2023-09-01T00:00:00Z"),
    contact: { // nested object
      phone: "555-1234",
      email: "*Emails are not allowed*"
    }
  }

The teachers Collection contains two documents:

 {
    title: "Mr.",
    last_name: "Davis",
    subject: "Mathematics",
    tenure: true,
    phone: "555-5678"
    class_count: 5
  },
 
  {
    title: "Ms.",
    last_name: "Garcia",
    subject: "History",
    class_count: 4,
    office_hours: { // Nested Object
      day: "Wednesday",
      time: "2:00 PM - 3:00 PM"
    }
  }

Flexibility by Nature:

→ One user document might have a phone field, while the next one might not. This dynamic schema is the main reason developers choose MongoDB when dealing with data that evolves rapidly.

→ Therefore, MongoDB allows us to model data naturally, like objects (documents), and these objects are managed efficiently within logical groups called Collections.

The Essential Field: `_id`

→ Every Document must have a unique field named _id. This field serves as the Primary Key for the document (just like the unique ID in a SQL row).

→ If you don't specify an _id yourself when inserting data, MongoDB automatically generates one for you using a special data type called ObjectId. This ObjectId is guaranteed to be unique and even includes a timestamp, giving you information about the exact moment the document was created.

#4. Basic CRUD Operations:
(Assuming the MongoDB server is installed and mongosh is ready, we can now connect to the database to see how we create collections and documents)...

→ CRUD operations represent the four basic functions that underlie any persistent data storage system. In the context of a database (such as MongoDB, although the terms apply generally), these are the methods through which we interact with Documents within a Collection.

1. Create: `insertOne()`/`insertMany()`

→ The Create operation adds new documents into a collection.

insertOne(document): this method is used to insert a single new document into the collection.

  //this will add a new document in 
  //teachers collection
db.teachers.insertOne({
  title: "Mr.",
  last_name: "Jack",
  subject: "Geography",
  tenure: false,
  phone: "555-5566"
  class_count: 6
})

insertMany([document1, document2, ...]): this method is used to insert multiple documents into the collection with a single command. The argument is an array (list) of documents.

  //this will add two documents in 
  //students collection
db.students.insertMany([
   {
     first_name: "Lea",
     last_name: "Casy",
     age: 17,
     grade: 10
   },
   {
     first_name: "Jessy",
     last_name: "Smith",
     age: 18,
     grade: 10
   }
])

2. Read: `find()`/`findOne()`

→ The Read operation retrieves documents from the collection based on filtering criteria.

findOne(query): this method returns the first document that matches the filtering criteria specified in the query object. It is ideal when you know you need a single document, for example, based on a unique ID. If no query is specified (i.e., it is empty: {}), it returns the first document found in the collection.

//find the first student with name 'Lea'
db.students.findOne({ first_name: 'Lea' })

find(query): This method returns a cursor to all documents that match the specified filtering criteria. If the query object is empty ({}), it returns all documents in the collection.

//return all of documents from students colection
db.students.find()

//find all students whose age is greater than 16
db.students.find({ age: {$gt: 16 }})

3. Update: updateOne()/updateMany()

updateOne(filter, update_operation): Updates the first document that matches the given filter.

//finds the document with `first_name 'Jessy'` and
// sets the new grade to false.
db.students.updateOne(
  { first_name: 'Jessy' },
  { $set: { grade: false } }
)

updateMany(filter, update_operation): Updates all documents that match the given filter.

// It selects documents only if the field contact is absent. ($exists: false)
// and add the 'contact' object with default phone and email values.
db.students.updateMany(
  // 1. FILTER: Select documents missing the 'contact' field
  { contact: { $exists: false } }, 
  
  // 2. OPERATION: Use $set to add the new 'contact' object
  { 
    $set: { 
      "contact.phone": false, 
      "contact.email": "*Emails are not allowed*" 
    } 
  }
);

→ The Basic $set Operator: This is the most common update operator. It specifies which fields should be modified and what their new values are. It either replaces the value of an existing field or adds a new field if it doesn't exist.

Basic Syntax: In update calls, the update operation part will often look like this: {"$set": {"field_name": "new_value", "another_field": 123}}.

4.Delete: deleteOne()/deleteMany

→ The Delete operation removes documents from the collection.
deleteOne(filter): deletes the first document that matches the specified filter.

// delete the first student found whose first name is 'Lea'.
db.students.deleteOne({ first_name: "Lea" })

deleteMany(filter): deletes all documents that match the specified filter.

// delete all students who are currently in grade 10.
db.students.deleteMany({ grade: 10 });

Caution❗If you use deleteMany({}) (an empty filter), you will delete all documents in the collection. Always be careful when using delete operations.

Delete a colection: db.students.drop(),
Delete a database: first select the database use school, then we delete it db.dropDatabase()

Conclusion ❤️

While I haven't delved into the specifics of working tools, my goal for this first article was to provide a short but solid introduction to the fundamental of MongoDB.

There is a lot to explore in the world of MongoDB, but I believe these basics are essential. If you understand how data saving and retrieval work in localStorage, you can draw a useful analogy to better grasp collections within a database.
I will be back soon with more detailed posts on the subject, further deepening the concepts we've learned. I hope this introductory guide was helpful!

If you enjoyed this article, learned something new, or if it brought back a smile thinking about your first attempts in MongoDB, don't forget to like the post or leave a comment! Thank you so much for reading and spending your valuable time with this article. I look forward to connecting with you in the next one! Happy codding!

Local MongoDB like database in flutter

Mon, 05 May 2025 17:07:35 +0000

FlutterDB

FlutterDB is a lightweight, SQLite-based document database for Flutter applications. It provides a MongoDB-like API for managing collections and documents while leveraging the performance benefits of SQLite.

Why do I build this?

So I am using MongoDB in my NodeJS application. Now I am building a chat application in Flutter, and for rendering performance, I thought I should store the messages locally and render a sliding window on it. Still, I have no luck finding a good local database that is flexible enough and also fast enough. And so I thought, why not build a MongoDB-like database in Flutter, which gets the job done. It's not a full-fledged MongoDB running locally, but it is similar to it, and most of the common tasks are going very well, as I tested.

Features

Document-oriented database with collections and documents
MongoDB-style query syntax and operations
Built on top of SQLite for performance and reliability
Unique ID generation for documents
Support for complex queries, updates, and aggregations
Batch operations for better performance

Installation

Add the following dependencies to your pubspec.yaml:

dependencies:
  flutterdb: ^0.0.1

Getting Started

Import the package:

import 'package:flutterdb/flutterdb.dart';

Initialize the database:

final db = FlutterDB();

Basic Usage

Working with Collections

// Create or get a collection
final users = await db.collection('users');

// List all collections
final collections = await db.listCollections();

// Drop a collection
await db.dropCollection('users');

CRUD Operations

Create

// Insert a single document
final userId = await users.insert({
  'name': 'John Doe',
  'email': '*Emails are not allowed*',
  'age': 30
});

// Insert multiple documents
final ids = await users.insertMany([
  {
    'name': 'Jane Smith',
    'email': '*Emails are not allowed*',
    'age': 25
  },
  {
    'name': 'Bob Johnson',
    'email': '*Emails are not allowed*',
    'age': 40
  }
]);

Read

// Find all documents in a collection
final allUsers = await users.find({});

// Find documents with a simple query
final adults = await users.find({'age': {'$gte': 18}});

// Find a document by ID
final user = await users.findById(userId);

// Count documents
final userCount = await users.count({'age': {'$gt': 30}});

Update

// Update a document by ID
await users.updateById(userId, {'status': 'active'});

// Update multiple documents
final updatedCount = await users.updateMany(
  {'age': {'$lt': 30}},
  {'status': 'young'}
);

Delete

// Delete a document by ID
await users.deleteById(userId);

// Delete multiple documents
final deletedCount = await users.deleteMany({'status': 'inactive'});

Advanced Queries

Comparison Operators

// Greater than
await users.find({'age': {'$gt': 30}});

// Less than or equal
await users.find({'age': {'$lte': 25}});

// Not equal
await users.find({'status': {'$ne': 'inactive'}});

// In array
await users.find({'role': {'$in': ['admin', 'moderator']}});

Logical Operators

// AND
await users.find({
  '$and': [
    {'age': {'$gte': 18}},
    {'status': 'active'}
  ]
});

// OR
await users.find({
  '$or': [
    {'role': 'admin'},
    {'permissions': {'$in': ['write', 'delete']}}
  ]
});

// NOR
await users.find({
  '$nor': [
    {'status': 'banned'},
    {'role': 'guest'}
  ]
});

Text Search

// Regex search
await users.find({'name': {'$regex': '^Jo'}});

// Simple text search
await users.find({'bio': {'$like': 'flutter developer'}});

Aggregation

final results = await users.aggregate([
  {'$match': {'status': 'active'}},
  {'$sort': {'age': -1}},
  {'$skip': 10},
  {'$limit': 20},
  {'$project': {
          'name': 1,
          'age': 1,
          'city': 1,
          '_id': 0,
               }
  },
  {'$group': {
          '_id': '\$city',
          'count': {'\$sum': 1}
        }
  },
]);

Best Practices

Performance Optimization

Use batch operations for multiple insertions or updates:

await users.insertMany(manyUsers); // Better than inserting one by one

Keep document size reasonable:
- Large documents can slow down performance
- Consider storing large binary data (images, files) separately
Create indexes for frequently queried fields:
- Currently, FlutterDB automatically indexes collection names
- Future versions may support custom indexes
Use appropriate queries:
- Finding by ID is faster than complex queries
- Limit results when possible using $limit

Data Structure

Use consistent schemas:
- While FlutterDB is schemaless, consistent document structures improve code maintainability
Choose good document IDs:
- Let FlutterDB generate IDs unless you have specific requirements
- Custom IDs should be unique and consistent
Handle relationships thoughtfully:
- For one-to-many relationships, consider embedding related data or using references

Error Handling

Always implement error handling:

try {
  await users.insert({'name': 'John'});
} catch (e) {
  print('Error inserting document: $e');
  // Handle error appropriately
}

Limitations

Not a full MongoDB replacement:
- Limited subset of MongoDB query operators
- Some complex aggregation operations not supported
Performance with large datasets:
- Best suited for mobile apps with moderate data size
- Consider pagination or limiting queries for large collections
No built-in encryption:
- Data is stored in plain text
- Consider additional encryption for sensitive data
Limited indexing options:
- Custom indexes not yet supported
- Consider query performance for large collections
No network synchronization:
- Local database only
- Implement your own sync solution if needed

Complete Example

import 'package:flutter/material.dart';
import 'package:flutterdb/flutterdb.dart';

class UserRepository {
  final FlutterDB _db = FlutterDB();
  late Future<Collection> _users;

  UserRepository() {
    _users = _db.collection('users');
  }

  Future<String> addUser(String name, String email, int age) async {
    final collection = await _users;
    return await collection.insert({
      'name': name,
      'email': email,
      'age': age,
      'createdAt': DateTime.now().toIso8601String(),
    });
  }

  Future<List<Map<String, dynamic>>> getAdultUsers() async {
    final collection = await _users;
    return await collection.find({
      'age': {'$gte': 18},
    });
  }

  Future<void> updateUserStatus(String id, String status) async {
    final collection = await _users;
    await collection.updateById(id, {
      'status': status,
      'updatedAt': DateTime.now().toIso8601String(),
    });
  }
}

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  
  final userRepo = UserRepository();
  
  // Add a user
  final userId = await userRepo.addUser('Alice', '*Emails are not allowed*', 28);
  print('Added user with ID: $userId');
  
  // Update user status
  await userRepo.updateUserStatus(userId, 'premium');
  
  // Get all adult users
  final adults = await userRepo.getAdultUsers();
  for (var user in adults) {
    print('User: ${user['name']}, Age: ${user['age']}, Status: ${user['status']}');
  }
}

Future Enhancements

Custom indexing support
More aggregation pipeline operators
Full-text search capabilities
Data encryption options
Schema validation
Observable queries
Migration support

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the GNU AGPLv3 License - see the LICENSE file for details.

If you are searching for this keywords
"offline-first Flutter apps"
"local storage in Flutter"
"best local database Flutter"

your search ends here
I'd love your thoughts. Try FlutterDB in your next project and let me know how it goes! You can ⭐️ the GitHub repo or open an issue if you hit any snags.

Build a Full-Stack Video Streaming App with React.js, Node.js, Next.js, MongoDB, Bunny CDN, and Material UI

Sat, 29 Mar 2025 09:22:42 +0000

The rise of video streaming platforms like YouTube, Netflix, and Twitch has transformed how we consume media. But what if you could build your own? In this comprehensive guide, we’ll walk you through creating a full-stack video streaming application using a modern tech stack: React.js, Node.js, Next.js, MongoDB, Bunny CDN, and Material UI (MUI). By the end, you’ll have a feature-rich app that supports video uploads, streaming, user interaction, and more—perfect for personal projects, learning, or even launching your own platform. Let’s dive into the process!

In this tutorial, we’ll create a powerful, full-stack video streaming application using React.js for dynamic, reusable components, Node.js for a reliable and scalable backend, Next.js for seamless server-side rendering and routing, MongoDB for efficient and flexible data management, Bunny CDN for lightning-fast video delivery, and Material UI (MUI) for a sleek, professional user interface. By the end of this journey, you’ll have a fully functional streaming app packed with modern, cutting-edge features—ready to rival platforms like YouTube or Netflix. Let’s get started!

Key Features You’ll Build:

✅ HLS Video Streaming: Deliver smooth, adaptive video playback across all devices.

✅ Debounced Search Functionality: Enhance search performance with efficient, user-friendly input handling.

✅ Data Fetching & Pagination with SWR: Seamlessly fetch video data with infinite scrolling and caching.

✅ Notification Display: Keep users engaged with real-time alerts and updates.

✅ MUI Toolpad Dashboard Design: Build a robust admin dashboard using Material UI’s advanced tools.

✅ Bookmark: Allow users to save their favorite videos effortlessly.

✅ Like & Unlike: Add interactive like/unlike features to boost engagement.

✅ Save Video History & Last Watched Time: Track user viewing progress and history with ease.

✅ Comment: Enable users to leave comments and interact with content.

✅ MongoDB Modeling: Design an optimized database schema for videos, users, and more.

✅ Upload & Publish Videos in Different Resolutions: Process and store videos in multiple quality levels.

✅ Securely Download Videos: Provide safe, authenticated video downloads for offline viewing.

✅ Share Videos to Different Platforms: Integrate social sharing for platforms like Twitter and Facebook.

✅ Implement PWA: Transform your app into a Progressive Web App for a native-like experience.

✅ SEO-Friendly: Optimize for search engines using Next.js SSR and metadata.

✅ Check Video Statistics: Display analytics like views, likes, and watch time.

✅ Stream in Different Resolutions & Quality: Let users select their preferred playback quality.

✅ Check Trending & Subscribed Channels: Showcase trending content and user subscriptions.

Why This Tech Stack?

Before we start, let’s break down why this combination of tools is ideal for a video streaming app:

React.js: Powers dynamic, reusable UI components for a responsive and interactive frontend.
Next.js: Enhances the app with server-side rendering (SSR), static site generation, and built-in routing for performance and SEO.
MongoDB: Offers a flexible, NoSQL database for storing video metadata, user profiles, and interactions.
Bunny CDN: Ensures ultra-fast video delivery with global content distribution and HLS streaming support.
Material UI (MUI): Delivers a polished, professional design system and tools like MUI Toolpad for rapid UI development.

Full Video Implementation

A video is worth a thousand words. Click here to watch

Conclusion

By the end of this project, you’ll have a full-stack video streaming app that’s not just functional but packed with modern features like HLS streaming & PWA support. Whether you’re building for fun, learning, or launching a startup, this app is a solid foundation. Experiment with additional features like live streaming or analytics to take it even further. Happy coding, and enjoy your YouTube-inspired creation!

4 ways to query GeoJSON data in MongoDB

Tue, 25 Mar 2025 23:51:49 +0000

Have you ever heard of GeoJSON? If not, let me explain it to you.
GeoJSON is a format for encoding a variety of geographic data structures created and published in 2008.

Since 2015, it has an official specification created by its original authors with the collaboration of the Internet Engineering Task Force (IETF).

According to this specification:

GeoJSON is a geospatial data interchange format based on JavaScript Object Notation (JSON). It defines several types of JSON objects and the manner in which they are combined to represent data about geographic features, their properties, and their spatial extents. GeoJSON uses a geographic coordinate reference system, World Geodetic System 1984, and units of decimal degrees.

It is a format widely used across JSON-based applications to read, manipulate and compare geospatial data.

Several third-party map libraries provide GeoJSON support for front-end applications, some of them are:

Leaflet
Openlayers
Google Maps

Considering that now you know a little bit more about GeoJSON, let´s move to the manipulation bit.
When building maps for a web application, it is common to deal with something we call "Layers". According to Google:

Layers are objects on the map that consist of one or more separate items but are manipulated as a single unit. Layers generally reflect collections of objects that you add on top of the map to designate a common association.

A Layer can have different formats and one of them is GeoJSON.

So, when working with more than one layer in a map, it is common to perform operations between them to determine their geospatial relationship within a context. By making use of geospatial operations, it is possible to determine, for example, if:

i) a Point lies within a Polygon;
ii) a Polygon intersects another Polygon;
iii) a Line crosses a circle;
iv) a Point is part of a Line;

But, depending on the size of the involved layers, manipulating or comparing them directly with JavaScript, in the browser, might cause performance issues since the memory usage tends to increase even more if they need to be rendered on a map.

A good solution for this case is to perform any required geospatial operation directly in the database. Most of the database engines provide functions to query GeoJSON and one of them is MongoDB which, by the way, has been pretty optimized for spatial operations since version 3.2.

These are some of the most basic geospatial structures that MongoDB supports:

Point

{
  "type": "Point",
  "coordinates": [
    -42.53700256347656,
    -22.28496664336935
   ]
}

Linestring

{
  "type": "LineString",
  "coordinates": [
    [
      -43.17626953125,
      -22.91792293614603
    ],
    [
      -40.2978515625,
      -20.262197124246534
    ]
  ]
}

Polygon

{
  "type": "Polygon",
  "coordinates": [
    [
      [
        -42.94006347656249,
        -22.649502094242195
      ],
      [
        -42.16552734375,
        -22.649502094242195
      ],
      [
        -42.16552734375,
        -21.968519331082298
      ],
      [
        -42.94006347656249,
        -21.968519331082298
      ],
      [
        -42.94006347656249,
        -22.649502094242195
      ]
    ]
  ]
}

Geometry Collection

{
  type: "GeometryCollection",
  geometries: [
    {
      "type": "Point",
      "coordinates": [
        -42.53700256347656,
        -22.28496664336935
      ]
    },
    {
      "type": "LineString",
      "coordinates": [
        [
          -43.17626953125,
          -22.91792293614603
        ],
        [
          -40.2978515625,
          -20.262197124246534
        ]
      ]
    },
    {
      "type": "Polygon",
      "coordinates": [
        [
          [
            -42.94006347656249,
            -22.649502094242195
          ],
          [
            -42.16552734375,
            -22.649502094242195
          ],
          [
            -42.16552734375,
            -21.968519331082298
          ],
          [
            -42.94006347656249,
            -21.968519331082298
          ],
          [
            -42.94006347656249,
            -22.649502094242195
          ]
        ]
      ]
    }
  ]
}

Here you can find the mentioned examples plotted in a map using geojson.io.

Now, I'm going to show you some basic operations that can be performed directly in a Mongo database assuming that there is already a collection named geospatial with the proper indexes set up.

To ease the query implementation, I recommend setting the GeoJSON object inside a property of your document. In the following example, the property "location" represents the geometry:

{
  "$id": "342j3ug4kj23b4k234"
  "location": {
    "type": "Point",
    "coordinates": [ -22, -42 ]
  }
}

1. Finding geometries that intersect with the given Point

db.geospatial.find({
  location: {
    $geoIntersects: {
     $geometry: {
       type: "Point",
       coordinates: [
         -42.53700256347656,
         -22.28496664336935
        ]
      }
    }
  }
})

This query will retrieve all geometries with which the given Point intersects. This example shows a query passing a Point as a parameter, but it supports any GeoJSON Object

2. Finding all geometries that lie within a given geometry

db.geospatial.find({
  location: {
    $geoWithin: {
      $geometry: {
        "type": "Polygon",
        "coordinates": [
          [
            [
              -42.94006347656249,
              -22.649502094242195
            ],
            [
              -42.16552734375,
              -22.649502094242195
            ],
            [
              -42.16552734375,
              -21.968519331082298
            ],
            [
              -42.94006347656249,
              -21.968519331082298
            ],
            [
              -42.94006347656249,
              -22.649502094242195
            ]
          ]
        ]
      }
  }
    }
})

This query will retrieve all geometries that are inside the given Polygon and it is useful to look for all points that are placed within an area. For example: let's say you'd like to find all restaurants in a given city.

3. Finding geometries within a number of miles from a given point

db.geospatial.find({
  location: {
    $geoWithin: {
      $centerSphere: [
        [ -42.53700256347656, -22.28496664336935 ],
        5 / 3963.2
      ]
    }
  }
})

Using $centerSphere, MongoDB will "draw" a circle with a pre-defined size from a given point. In this case, the query will retrieve all geometries that are up to 5 miles to the coordinates provided.
It is important to mention that, in this case, the geometries will be retrieved unordered.

4. Finding ordered geometries within a number of miles from a given point

const METERS_PER_MILE = 1609.34

db.geospatial.find({
  location: {
    $nearSphere: {
      $geometry: {
        type: "Point",
        coordinates: [ -73.93414657, 40.82302903 ]
      },
      $maxDistance: 5 * 1609.34
    }
  }
})

This query will retrieve the same result as the previous one, but order from the nearest to the farthest.
Check this page for more information on distance unit conversion.

GeoJSON is the most used format to represent Geospatial data in web applications. It is fully supported by the most popular third-party map rendering libraries and database engines.

This article demonstrated some basic concepts and how to implement some of the spatial operations provided by MongoDB.

I hope you've liked it.
Please, share and comment... any feedback is welcome!

Best Databases to Use with the MERN Stack Apart from MongoDB

Tue, 11 Mar 2025 03:19:58 +0000

The MERN stack (MongoDB, Express, React, Node.js) has become one of the most popular choices for modern web development due to its flexibility, scalability, and the ease with which the technologies work together. MongoDB is the default database for many MERN stack developers, primarily because it is a NoSQL database and integrates seamlessly with Node.js. However, MongoDB isn’t the only option available for building a robust, scalable backend. Depending on your project’s requirements, you might prefer to use a different database.

In this article, we will explore the best databases to use with the MERN stack, apart from MongoDB. Each database has its unique strengths and can be an excellent choice for specific use cases, from relational data management to high scalability and real-time applications. Whether you're building a large-scale enterprise application or a small personal project, there’s a database that will suit your needs.

1. PostgreSQL: A Powerful Relational Database

PostgreSQL is an open-source, object-relational database system that is widely regarded for its stability, extensibility, and support for advanced SQL features. If your application requires complex queries, a high level of data integrity, or sophisticated transaction handling, PostgreSQL is an excellent choice. It is fully ACID-compliant, which ensures that transactions are processed reliably, even in the event of a system crash.

PostgreSQL is not only known for its SQL features but also its ability to handle JSON data types through its JSONB feature. This gives developers the flexibility to combine the structure of SQL with the unstructured nature of NoSQL, making it ideal for hybrid applications. If your project involves extensive data relationships, PostgreSQL's support for foreign keys, joins, and transactions ensures a solid foundation for building complex, data-driven applications.

When integrating PostgreSQL with a MERN stack, you can use Node.js libraries such as pg or ORM frameworks like Sequelize or TypeORM for a more structured approach. These tools allow you to interact with PostgreSQL using JavaScript, making it easy to manage the database and implement queries directly within your Node.js backend.

2. MySQL: A Highly Efficient Relational Database

MySQL is another widely used relational database that has been a staple in the development world for decades. Known for its speed and reliability, MySQL is a strong contender for any application that requires structured data storage and SQL querying. It is particularly useful for applications that handle a large number of read and write operations, such as e-commerce platforms, content management systems, and business applications.

One of MySQL's standout features is its performance, especially when it comes to handling read-heavy applications. For projects that involve a large number of concurrent users or need fast access to data, MySQL provides excellent scalability options through indexing and query optimization techniques.

MySQL also supports replication, which means you can set up master-slave configurations for load balancing and increased reliability. This makes it an excellent choice for building high-traffic applications. If your project relies on relational data and complex queries, MySQL is a solid option for your MERN stack backend.

Integrating MySQL with MERN can be done using libraries like mysql2 or using ORM tools like Sequelize or TypeORM, which allow developers to interact with MySQL using JavaScript and simplify the management of data relationships.

3. SQLite: Lightweight and Serverless

SQLite is a lightweight, file-based database that doesn’t require a dedicated server. It’s an excellent choice for small-scale applications, mobile apps, or for use in development and testing environments. While SQLite isn’t suitable for applications that need to scale horizontally or handle heavy concurrent traffic, it is ideal for situations where simplicity and ease of use are paramount.

SQLite is embedded directly into the application, making it easy to set up and use without the overhead of managing a separate database server. It is commonly used in mobile applications or small web apps that don’t require a full-fledged database management system.

When working with SQLite in the MERN stack, you can use the sqlite3 package in Node.js, or you can opt for an ORM like Sequelize, which allows you to define models and perform database operations seamlessly within your application.

4. Cassandra: A NoSQL Database for High Scalability

Cassandra is a distributed NoSQL database designed to handle huge amounts of data across many servers. It is built to provide high availability, scalability, and fault tolerance, making it ideal for applications that need to handle large volumes of data and ensure uptime even during network failures or server outages. If your application requires a highly scalable, fault-tolerant solution, Cassandra might be the right choice.

Unlike MongoDB, which stores documents, Cassandra uses a column-family store to organize data. This makes it particularly well-suited for time-series data, real-time analytics, or applications that require fast writes and large data sets. Cassandra is used by companies like Netflix, Instagram, and Spotify for managing their massive datasets at scale.

Integrating Cassandra with your MERN stack can be done using the cassandra-driver for Node.js. This driver allows you to connect to Cassandra, perform queries, and manage data within your backend.

5. Redis: A High-Performance In-Memory Store

Redis is an in-memory data store that can be used as a cache, session store, or for real-time data processing. While Redis isn’t typically used as a primary database for storing persistent data, it’s highly effective for scenarios where low-latency access to data is critical. Redis excels at handling high-throughput workloads and can store a wide range of data types, including strings, hashes, lists, sets, and sorted sets.

Redis is commonly used in applications that require fast data retrieval, such as caching frequently accessed data, managing user sessions, or handling real-time data for messaging or notification systems. It can also be used to store application state or for pub/sub messaging between services.

To integrate Redis with MERN, you can use the ioredis or redis Node.js clients. These libraries make it easy to connect Redis to your Node.js backend and implement caching, sessions, and real-time features.

6. Firebase Firestore: Real-Time NoSQL Database

Firebase Firestore is a cloud-based NoSQL database developed by Google. It provides real-time data synchronization across multiple devices and offline data persistence, which makes it ideal for mobile apps or applications that require real-time updates. Firestore is part of the broader Firebase ecosystem, which includes authentication, hosting, and cloud functions, making it a powerful tool for building full-stack applications.

Firestore is particularly useful for applications that require automatic syncing of data across devices, such as collaborative apps, chat applications, or apps with live updates. The real-time features in Firestore allow for seamless user experiences, where data updates reflect instantly without needing to refresh the app.

Integration with MERN is straightforward, as Firebase provides an official SDK that can be easily used in your Node.js backend. Firebase also offers SDKs for React, which makes it simple to integrate Firestore into the front end of your MERN stack application.

7. CouchDB: A Document-Based NoSQL Database

CouchDB is another document-based NoSQL database that uses JSON for data storage. CouchDB is designed with replication and synchronization in mind, making it an excellent choice for apps that need offline capabilities or peer-to-peer data syncing. It is known for its simplicity, reliability, and ease of use. CouchDB supports MapReduce for views, which can be used to aggregate data.

Similar to MongoDB, CouchDB stores JSON documents but offers a unique set of features for working with distributed data. If your application needs easy replication across multiple nodes or offline data access, CouchDB might be an ideal fit.

To integrate CouchDB into your MERN stack, you can use libraries like nano or pouchdb, which make it easy to interact with the database from your Node.js application.

8. Neo4j: A Graph Database for Connected Data

Neo4j is a graph database that is specifically designed to store and query highly connected data. It’s an ideal choice for applications that need to work with complex relationships, such as social networks, recommendation engines, and fraud detection systems. Instead of using rows and columns, Neo4j stores data as nodes and edges, which are linked to each other through relationships.

If your MERN stack application requires handling of highly interconnected data, such as modeling relationships between users, products, and transactions, Neo4j is a powerful solution. Neo4j’s Cypher query language allows you to query graph data intuitively and efficiently, making it a great choice for applications that need to explore complex relationships between entities.

Neo4j can be integrated with MERN using the official neo4j-driver for Node.js, which allows you to query the graph database directly from your backend and incorporate it into your application.

MongoDB: A Beginner's Guide to NoSQL Databases

Sun, 02 Feb 2025 09:56:37 +0000

What is MongoDB?

MongoDB is a NoSQL, document-based database that stores data in a JSON-like format called BSON (Binary JSON). Unlike SQL databases, MongoDB does not require a fixed schema. This flexibility makes it great for applications that need to scale quickly and manage different types of data.

Example:

Imagine a social media app storing user information. Each user may have different attributes: some may have phone numbers, others might not. MongoDB allows you to store this data without needing to define a rigid structure, unlike traditional SQL databases.

What MongoDB is Not

MongoDB is not a relational database.

While SQL databases work best for applications that require strict relationships between data (like financial systems or traditional enterprise applications), MongoDB is best for handling unstructured data or applications that require quick scalability, like real-time analytics, content management systems, and IoT platforms.

Why You Should Learn MongoDB

As the demand for modern, data-driven applications continues to grow, MongoDB is becoming more relevant. Here's why learning MongoDB in 2025 is a good idea:

Growing Demand: More companies are adopting NoSQL databases for their flexibility and scalability.
Ease of Use: MongoDB's document-based structure is easier to work with than SQL's table-based structure, making it ideal for beginners.
Big Data and Cloud-Friendly: MongoDB is widely used for managing big data and working in cloud environments.

MongoDB vs SQL

One of the most common comparisons is between MongoDB and traditional SQL databases.

- Schema: SQL requires a fixed schema (tables, rows, columns), while MongoDB's schema-less design allows you to store different types of data without predefined structures.

- Scaling: MongoDB is built to scale horizontally (across multiple servers), while SQL databases often scale vertically (increasing server power).

- Data Storage: SQL uses tables to store data, while MongoDB stores data in collections of documents, which are more flexible.

Advantages of Using MongoDB

Scalability: MongoDB can handle large amounts of data, scaling easily as your application grows.
Flexibility: It allows for storing various data types, from simple records to complex documents with nested arrays and objects.
High Performance: MongoDB offers fast data retrieval due to its document-based structure.
Real-time Analytics: MongoDB is ideal for applications that require real-time processing, like analytics platforms.

MongoDB Drawbacks

While MongoDB has many advantages, it's important to note a few drawbacks:

Lack of Complex Transactions: While MongoDB supports basic transactions, it doesn't handle complex multi-table transactions as efficiently as SQL databases.
Consistency: In certain use cases, MongoDB might sacrifice strict data consistency (ACID properties) for performance and scalability. This could be a concern for systems that require precise consistency, like banking systems.

When to Use MongoDB

You should consider using MongoDB when:

You are building a project that requires scalability and flexibility, such as social networks, content management systems, or real-time analytics platforms.
Your application deals with unstructured or semi-structured data (e.g., logs, user-generated content, etc.).
You need fast development cycles and agile methodologies where schema design might change often.

When Not to Use MongoDB

MongoDB is not ideal when:

Your project requires complex transactions (e.g., banking applications or systems requiring multi-step business logic).
Your data needs to follow strict relationships and constraints (e.g., inventory systems, financial applications, or CRM systems).
Consistency is more important than availability or partition tolerance (known as the CAP theorem).

Benefits of MongoDB:

Schema Flexibility: Ideal for applications that handle varying types of data.
Horizontal Scalability: You can scale your app seamlessly with sharding.
High Availability: MongoDB supports replication for fault tolerance.
Faster Development: You can iterate faster with MongoDB's flexible data structure.

Despite the benefits, MongoDB may not be suitable for all use cases.

Learning Curve: Although MongoDB is beginner-friendly, getting familiar with its advanced features (like sharding and replication) can take time.
Data Integrity: MongoDB does not enforce data integrity as strictly as relational databases, so it's important to plan for handling data consistency issues in your application.

Conclusion

MongoDB is an excellent choice for modern applications that need scalability, flexibility, and fast development. While it's not suitable for all use cases, especially those requiring complex transactions or rigid data structures, it is a powerful tool for developers working on dynamic applications. By understanding the strengths and weaknesses of MongoDB, you can make an informed decision about whether it's the right tool for your project.

Have you worked with MongoDB before? What was your experience? Share your thoughts in the comments below, or feel free to ask questions. For more beginner tutorials, check out my related articles!

Glossary

NoSQL: A type of database designed to handle unstructured or semi-structured data.
BSON: A binary format of JSON used by MongoDB for storing documents.
Sharding: A method of distributing data across multiple servers to ensure scalability.
Replication: The process of copying data across multiple servers to ensure high availability.

Coder Legion - Recent posts tagged mongodb

Your Pagination Is Showing Users the Same Item Twice (and Skipping Others). Here's Why Offset Breaks

The Code Everyone Writes

Why It Breaks

The Second Problem: It Gets Slower on Deep Pages

The Fix: Cursor Pagination

Step 1: Sort by something stable, with a tie-breaker

Step 2: Support the query with an index

Step 3: Query "after the cursor"

Step 4: Encode the cursor so clients do not depend on it

What You Give Up

When Offset Is Still Fine

The Rule

Your 'Pay Now' Button Can Charge Twice. Idempotency Keys Are the Fix Nobody Adds Until It Happens

The Endpoint That Looks Fine

Why This Is Hard to Catch

The Idea: Same Key, Same Result

A Working Version With Express and MongoDB

The Race Condition Most Examples Miss

Four Details That Matter

The Rule

Demystifying MongoDB Write Concern and Read Concern in Node.js

What Are Write Concern and Read Concern?

The Real-World Analogy: Certified Mail vs. Kitchen Whiteboards

Why It Matters in Daily Software Engineering

Implementation in Node.js

Key Takeaway

Your Mongoose pre('save') Hook Is Being Silently Skipped by Half Your Update Calls

The Setup

Where It Actually Breaks

Why This Specifically Gets Missed

The Fix: Use Query Middleware, Not Just Document Middleware

A More Defensive Alternative: Standardize on One Update Path

The Rule

🧠 **Coder Legion Dev Log — Building the MyZubster Metaverse from the infrastructure up**

Identity and session model

Realtime authorization

Room lifecycle

Frontend state hardening

Auth and observability testing

Deployment discipline

Current architecture direction

Skills this work is exercising

CoderLegion #SoftwareEngineering #React #NodeJS #MongoDB #Realtime #SystemDesign #DevOps #WebDevelopment #Metaverse #OpenSource

Your Zod Schema's .optional() Might Be Silently Rejecting Valid Data From MongoDB

What Each One Actually Validates

Where This Actually Breaks With MongoDB Specifically

The Fix: Match the Schema to What the Data Actually Does

Why This Doesn't Show Up Until Real Data Hits It

A Quick Way to Check Which One You Actually Need

The Actual Rule

generateMetadata and Your Page Component Are Probably Querying the Same Data Twice

The Setup Almost Every Dynamic Page Has

Why This Specific Case Matters More Than It Might Seem

The Fix: Wrap the Query in React's cache()

Why This Is Worth Checking Specifically, Not Just Generally

A Quick Way to Verify This Is Actually Happening

Where This Doesn't Apply

The Actual Rule

How I Eliminated 19,494 Duplicate MongoDB Records (And What I'd Do Differently Today)

The Wake-Up Call

What Was Actually Happening

The Constraints

Designing the Fix

Migrating Without Downtime

Querying the New Schema

The Results

How I'd Do It Now

Lessons Learned

Building MyZubster: A Circular Marketplace Connected to a Social Metaverse

MyZubster #OpenSource #WebDevelopment #Metaverse #Marketplace #CircularEconomy #ReactJS #NodeJS #MongoDB #BuildInPublic

That .lean() Query Optimization Might Be Leaking Password Hashes in Your API Response

The Setup That Looks Like a Solid Security Pattern

Where This Quietly Breaks

Why This Is Genuinely Easy to Introduce

The Actual Fix: Never Rely on toJSON as Your Only Safety Net

The Broader Rule This Points To

Check Your Own API Routes

🚀 Building a Production Facebook Messenger → AI Pipeline with MyZubster & Zorgax

OpenSource #NodeJS #AI #FacebookMessenger #MetaDevelopers #Vercel #Webhooks #APIs #MongoDB #BuildInPublic #MyZubster #Zorgax

🧠 Coder Legion Dev Log — Building the MyZubster Metaverse from the infrastructure up

The Essential Field: `_id`

1. Create: `insertOne()`/`insertMany()`