Coder Legion - Recent posts tagged json

Three ways a jq one-liner lies to your bash script

Tue, 27 Oct 2026 13:00:00 +0000

Most "the filter looks right but the script misbehaves" bugs come down to three silent failure modes.

First, a missing -r. jq outputs JSON by default, so a string arrives with its quotes attached — your variable holds "web-01", quotes included, and a comparison against web-01 never matches. Any value headed into a shell variable, filename, or test wants -r.

Second, missing keys don't error. jq emits null, which lands in bash as a literal four-character string — non-empty, so [ -n ] guards pass and backup-null.tar.gz gets written. The // operator supplies a real fallback.

Third, select() compares exactly. A value that is almost right matches nothing, and nothing is not an error:

jq -r '.items[] | select(.active == true) | .name'

If the API sends "active": "true" as a string, that item is skipped. If you match "Web-01" against "web-01", zero elements come back. Either way jq prints nothing and exits 0, so the silence reads as success.

All three share the same fingerprint: exit code zero, plausible output, wrong result. The fastest way I've found to catch them before they ship is building the filter against the actual response: this interactive jq filter builder with a live in-browser preview lets you click through your real JSON to build the path, handles the -r, the // fallback and select() quoting for you, and shows exactly what the filter emits before anything runs.

The Trigger Swap That Broke Every Downstream Reference

Tue, 15 Sep 2026 13:15:42 +0000

I built this workflow — a lead intake automation for a made-up client, Ferrer Window Co. — with a Form Trigger first, even though I knew from the start it needed to end up on a Webhook. That wasn't the plan drifting. It was deliberate: I'd used webhooks before and Form Trigger less, so building with Form Trigger first let me see real data coming through without also holding webhook payload structure in my head at the same time. Get the logic right, then swap the trigger once everything else was solid.

And the logic did get solid. By the time I was ready to swap, I had a full chain working: an IF node filtering out junk leads (both phone and email empty), a Set node defaulting empty message fields to "No message provided," and Sheets and Slack both branching off that same cleaned data independently. Tested both edge cases — junk lead correctly stopped, empty message correctly defaulted. Everything passed.

Then I deleted the Form Trigger, added a Webhook node, and sent a test payload through Postman.

It broke immediately.

Not the logic — the logic was fine. What broke was every single reference to the data. With Form Trigger, the payload had come through flat:

{
  "Name": "Tarek",
  "Phone": "02-011-41104-582",
  "Email": "*Emails are not allowed*",
  "Service address": "Primary school st",
  "Message": "I want to request an order",
  "submittedAt": "2026-08-24T03:24:50.089-04:00",
  "formMode": "test"
}

With Webhook, everything I actually wanted was nested one level down, sitting alongside a pile of raw HTTP metadata I didn't need:

{
  "headers": {...},
  "body": {
    "Name": "Tarek",
    "Phone": "02-011-41104-582",
    ...
  },
  "webhookUrl": "...",
  "executionMode": "test"
}

Every reference I'd written — the IF node's condition, the Set node's field mappings — pointed at $json.Name, $json.Phone, and so on. None of it was wrong, exactly. It was just pointed at a shape of data that no longer existed. The whole workflow's reasoning was intact and now unreachable.

So I went back through it node by node — the IF node's conditions, the Set node's mappings — and repointed every one from $json.X to $json.body.X. Nothing about what the workflow decided changed. Only where it looked for the thing it was deciding about.

Once that was done, I didn't just assume it worked because the references compiled. I re-ran both edge cases through the webhook path — both phone and email empty, and an empty message field — to confirm the filter still stopped junk leads and the default still kicked in, now that the data was arriving through a different shape entirely.

The part I'd flag for anyone doing this same swap later: it's not a dramatic failure. Nothing throws a loud error that points you at the fix. The workflow just quietly stops seeing the data it expects, because the trigger node changed the shape of what "the data" even means. If you built your logic against one trigger's payload structure, swapping the trigger isn't a drop-in replacement — it's a second pass through everything downstream, checking what each node thinks it's looking at.

Why Your LLM Still Returns Broken JSON

Mon, 07 Sep 2026 16:12:41 +0000

Most of us here have shipped something that calls a model and then spends half its code cleaning up whatever came back. This is the writeup I wish I had the first time I wired an LLM into a real pipeline, so I am leaving it here for anyone about to do the same.

The Three Workarounds We All Wrote First

Prompt engineering: tell the model to respond in JSON with these exact fields, and hope. That holds roughly 85 to 95 percent of the time depending on the model, which sounds fine until you are handling real traffic and it fails a few hundred times a day.

Post processing: regex parsers, JSON repair libraries, retry loops. Adds latency and a whole class of bugs that have nothing to do with your actual business logic.

Function calling: works well for invoking a tool, semantically awkward when all you want is data back rather than a call made.

What Constrained Decoding Actually Does

At each generation step the model produces a probability distribution over its entire vocabulary. Constrained decoding masks out every token that would take the output off a valid path through your schema, before sampling happens.

If your schema says sentiment must be one of positive, negative or neutral, the model cannot emit "somewhat positive". Those tokens are not available to it. That is enforced in the inference stack rather than in your prompt, which is why the guarantee is absolute instead of probabilistic.

JSON Mode Is Not Schema Enforcement

This one trips people up constantly. JSON mode guarantees the response parses. It does not guarantee the response has your fields. You can ask for a sentiment and a confidence score and get back {"answer": "yes"}, which is perfectly valid JSON and useless to your code.

Schema enforcement guarantees conformance to the contract: your fields, your types, your enums. If you are shipping to production, that is the one you want.

What It Still Does Not Give You

The schema promises that price is a number. It does not promise the number is positive, or that the email string is a real address. Schema enforcement removes structural errors and leaves semantic ones, so keep a validation layer such as Pydantic sitting on top of it.

Worth knowing before you design around it: most providers cap nesting around five levels, strict mode wants every property listed in the required array, and optional fields get expressed as anyOf with a null type.

The full breakdown, including provider by provider support and the migration path off JSON mode, is here: LLM structured output

If you have hit a case where schema enforcement made things worse rather than better, I would like to hear it. The failure modes are far less documented than the happy path.

JsonToolBox

Fri, 04 Sep 2026 22:58:29 +0000

JsonToolBox is a free, open-source JSON toolkit built for developers, testers, technical writers, and anyone who works with structured data.

It provides browser-based tools for common JSON tasks, including formatting and beautifying JSON, validating syntax, minifying output, viewing JSON in a structured tree, comparing two JSON documents, and converting JSON to other commonly used formats.

JsonToolBox is designed for fast, focused use. You can open a tool, paste your data, perform the required operation, and copy or download the result without creating an account. JSON processing is performed locally in your browser, so your JSON content is not uploaded to a remote server for processing.

Typical use cases include inspecting API responses, debugging configuration files, checking JSON before deployment, preparing examples for documentation, comparing payloads, and converting structured data for development workflows.

The project is open source and available under the MIT License. The complete source code and development information are available on GitHub.

Website: https://jsontoolbox.cc
Source code: https://github.com/kbmjj123/jsontoolbox.cc

Replay a Flight After It Lands: Historical Flight Tracker API

Tue, 01 Sep 2026 10:26:12 +0000

For developers, historical flight data opens up a more useful layer of analysis. Instead of working with only a final flight status, you can work with timestamped aircraft positions, altitude changes, speed values, route movement, ground status, and aircraft identifiers after the flight is completed.

The Aviation Edge Historical Flight Tracker API provides past flight movement data through a REST API. It returns recorded aircraft positions, altitude, direction, speed, timestamps, airline details, aircraft identifiers, route information, and final flight status.

In this article, we will look at what the API returns, how developers can work with the data, and what you can build with past flight tracker data.

What The API Returns

The Historical Flight Tracking API is built for retrieving detailed position data for past flights.

A response includes:

Departure and arrival airport codes
Scheduled departure and arrival times
Airline IATA and ICAO codes
Flight IATA and ICAO numbers
Aircraft ICAO24, aircraft type code, and registration number
Final flight status
Squawk information when available
A flightPositions array with recorded movement points

Each item in the flightPositions array represents a point in the aircraft’s journey.

Those position points include:

latitude
longitude
altitude
direction
horizontal speed
vertical speed
ground/airborne status
timestamp

That makes the API useful for rebuilding a past flight path, plotting aircraft movement on a map, or analyzing how a flight progressed from gate to gate.

Example Request

Here is an example request for historical tracking data for a flight from London Heathrow.

GET https://aviation-edge.com/v2/public/flight_track_history?key=YOUR_API_KEY&depIata=LHR&flightIata=BA203&depDate=2025-10-20

You can review the full parameter list and API specs in the Historical Flight Tracker API documentation.

Example Output

[
  {
    "aircraft": {
      "icao24": "40749B",
      "icaoCode": "B789",
      "regNumber": "G-ZBKR"
    },
    "airline": {
      "iataCode": "BA",
      "icaoCode": "BAW"
    },
    "arrival": {
      "iataNumber": "BOS",
      "icaoNumber": "KBOS",
      "scheduledTime": "2025-10-20T20:10:00.000"
    },
    "departure": {
      "iataNumber": "LHR",
      "icaoNumber": "EGLL",
      "scheduledTime": "2025-10-20T17:35:00.000"
    },
    "flight": {
      "iataNumber": "BA203",
      "icaoNumber": "BAW50G"
    },
    "flightPositions": [
      {
        "altitude": "0.00",
        "direction": "90.00",
        "horizontal_speed": "1.85",
        "isGround": 1,
        "latitude": "51.469400",
        "longitude": "-0.471700",
        "updated": 1760978610,
        "vspeed": "0.00"
      },
      {
        "altitude": "0.00",
        "direction": "90.00",
        "horizontal_speed": "0.00",
        "isGround": 1,
        "latitude": "51.469400",
        "longitude": "-0.471700",
        "updated": 1760978792,
        "vspeed": "0.00"
      },
...
...
      {
        "altitude": "0.00",
        "direction": "33.00",
        "horizontal_speed": "33.34",
        "isGround": 1,
        "latitude": "42.365300",
        "longitude": "-71.012400",
        "updated": 1761004300,
        "vspeed": "0.00"
      },
      {
        "altitude": "0.00",
        "direction": "303.00",
        "horizontal_speed": "11.11",
        "isGround": 1,
        "latitude": "42.373000",
        "longitude": "-71.018500",
        "updated": 1761004451,
        "vspeed": "0.00"
      }
    ],
    "status": "landed",
    "system": {
      "squawk": null,
      "updated": 1761004451
    }
  }
]

The flightPositions array is updated with recorded position points throughout the flight, typically at around 10-minute intervals, and added to the output as part of the historical flight track.

Field	Description
`latitude` / `longitude`	Aircraft position
`altitude`	Aircraft altitude
`direction`	Direction of travel
`horizontal_speed`	Horizontal speed
`vspeed`	Vertical speed
`isGround`	Whether the aircraft is on the ground
`updated`	Unix timestamp for that position point

Fetching Historical Flight Data in JavaScript

Here is a basic Axios example.

const axios = require("axios");

const API_KEY = "YOUR_API_KEY";

const endpoint = `https://aviation-edge.com/v2/public/flight_track_history?key=${API_KEY}&depIata=LHR&flightIata=BA203&depDate=2025-10-20`;

async function fetchHistoricalFlight() {
  try {
    const response = await axios.get(endpoint);
    return response.data;
  } catch (error) {
    console.error("Error fetching historical flight data:", error.message);
    return null;
  }
}

fetchHistoricalFlight().then((data) => {
  if (!data || data.length === 0) {
    console.log("No historical flight data found.");
    return;
  }

  console.log(data[0]);
});

Extracting the Flight Path

To plot the aircraft path on a map, you usually only need latitude and longitude from each position point. The output can be passed into a mapping library such as Leaflet, Mapbox, or Google Maps to draw the route.

function getFlightPath(flight) {
  return flight.flightPositions.map((point) => ({
    lat: Number(point.latitude),
    lng: Number(point.longitude),
    timestamp: point.updated
  }));
}

fetchHistoricalFlight().then((data) => {
  const flight = data[0];
  const path = getFlightPath(flight);

  console.log(path);
});

Finding the Max Altitude

A simple way to find the highest recorded altitude in the position array.

function getMaxAltitude(flight) {
  return Math.max(
    ...flight.flightPositions.map((point) => Number(point.altitude))
  );
}

fetchHistoricalFlight().then((data) => {
  const flight = data[0];

  console.log("Max altitude:", getMaxAltitude(flight));
});

Example Use Cases for Past Flight Track Data

Historical flight tracking data can be used in many aviation and data products.

Flight Replay: Use the latitude, longitude, altitude, speed, and timestamp fields to replay a flight on a map after it has landed and visualize a past flight.

Flight Analytics: Analyze altitude changes, speed changes, route shape, ground movement and airborne movement.

Airline or Airport Dashboards: Build dashboards showing past flight movement data for specific flights, airlines, airports, or aircraft.

Aircraft History Tools: Use aircraft registration number or ICAO24 filters to review past aircraft movement data for a specific aircraft.

Bringing Past Flights Back to Life - All With Data!

The Aviation Edge Historical Flight Tracking API is designed for developers who need more than a static flight record.

It gives you timestamped aircraft movement data that can be mapped, replayed, filtered, stored, and analyzed.

Whether you are building a flight replay tool, an aviation dashboard, an aircraft history feature, or a route analysis product, past flight tracking data gives you the raw movement layer behind a completed flight.

You can explore the Historical Flight Tracker API firsthand by getting your personal API key here.

A JSONPath engine that follows the spec and can't run code

Tue, 25 Aug 2026 11:53:27 +0000

I needed to pull a few fields out of some JSON. Nothing exotic, a couple of nested values and one filtered list. So I did the ordinary thing, reached for a JSONPath library, wrote a query, and moved on.

A week later I swapped that library for an unrelated reason, and the same query came back with different results. Not wrong exactly, just different. My $..author matched one extra node, a slice counted from the other end, and a filter that worked before now threw. Welcome to JSONPath, where for the better part of two decades every implementation agreed on the dollar sign and improvised the rest.

The standard nobody had

Stefan Gössner's original write-up from 2007 sketched the idea and left the corners to the reader. Everyone filled them in differently. Does $..book[0] give you the first book in each array or the first book overall? Do you write a filter [?(@.price)] or [?@.price]? What does a negative index do at the edge of an array? Pick two libraries and you could get two answers to each.

That changed in February 2024, when the IETF published RFC 9535 and wrote the rules down: the selectors, the filter grammar, the meaning of a missing value, with a compliance test suite so you can check yourself against them. JSONPath went from folklore to specification.

So I wanted a library that actually followed it. Reading the spec, I noticed the part that makes JSONPath worth using is also the part that keeps turning up in security advisories.

The filter that runs a shell

Filters are the reason you pick JSONPath over a hand-written loop. $..book[?@.price < 10] reads like a sentence and does real work: keep the nodes where the comparison holds.

Evaluating that comparison is where libraries cut corners. jsonpath-plus, the most-downloaded option on npm, built its filters on generated code. That works right up until part of the query comes from somewhere you don't control. At that point a filter is no longer a filter, it is a program running with your process's permissions, and that is exactly how it went: CVE-2024-21534 was a remote code execution through a crafted JSONPath, with more to follow when the first patch left a gap.

There is a quieter cost too. Generating code at runtime needs 'unsafe-eval' in your Content Security Policy, and if you have already shipped a strict CSP, granting that back undoes most of the reason you shipped it.

A scout that stays on the path

padvinder is my answer to both. The name is Dutch for "pathfinder", and also the word for a scout, which fit something whose entire job is to go find a node and come back without disturbing anything on the way. It implements RFC 9535 in roughly 2.75KB min+gzip with no dependencies, and it clears all 456 valid-selector cases in the official suite.

import { query, find } from 'padvinder';

const data = { store: { book: [
  { title: 'Sayings of the Century', price: 8.95, category: 'reference' },
  { title: 'Sword of Honour', price: 12.99, category: 'fiction' },
  { title: 'Moby Dick', price: 8.99, category: 'fiction' },
] } };

find('$..book[?@.price < 10].title', data);
// => ['Sayings of the Century', 'Moby Dick']

find runs a query once. For a path you will reuse, query compiles it and returns a runner you can point at different data:

const fiction = query('$.store.book[?@.category == "fiction"].title');
fiction(data); // => ['Sword of Honour', 'Moby Dick']

Hand it a broken path and it throws a SyntaxError at compile time, not partway through your data. A path that matches nothing returns [].

The parts of the spec worth carrying around

Selectors are the dependable half. If you have written JSONPath they hold no surprises: $ for the root, .name or ['name'] for a child, [0] and [-1] for indexes, [1:3:2] for a stepped slice, * for every child, .. for a recursive descent, and [0, 2] to union a handful of selectors.

Filters are where RFC 9535 spends its pages, and a few of its rulings are worth keeping in your head:

Existence is not truthiness. A bare [?@.discount] keeps every child that has a discount, even one set to null or 0. It asks whether the path exists, not whether the value is truthy.
A missing path is "nothing", not an error. In a comparison, [?@.a.b == 1] just fails to match when a is absent instead of throwing, so filters stay safe to write against ragged data.
<strong>==</strong> is deep equality. [?@.tags == ["new", "sale"]] compares structure, and none of JavaScript's loose coercion tags along.
<strong>$</strong> is still the root inside a filter. A child can be measured against something elsewhere in the document: [?@.price > $.store.bicycle.price].

Five functions ship built in, taken straight from the RFC: length(), count(), value(), and the regex tests match() for a whole string and search() for a substring. For anything else you pass your own and call it with @ as the current node:

find('$..book[?discounted(@)].title', data, { discounted: b => b.price < 9 });

That registry is the single place executable code enters, and it enters from your side, by name. Whoever writes the query chooses which of your functions to call, never what it does.

The part I tried to borrow

padvinder started out leaning on xprsn. A JSONPath filter is an expression over the current node, so my first version handed each [?...] straight to xprsn and let it do the evaluating. The easy queries passed. The spec did not.

RFC 9535 filters only look like ordinary expressions. Their operands are queries as often as values, so @.a.b is a small path that runs and may match nothing at all. A bare @.discount is an existence test, not a truthiness check. Equality is structural. A missing singular query turns into a special "nothing" that compares equal to no value on earth. A general expression evaluator has no reason to think in any of those terms, and bending xprsn to fit would have meant loading it with JSONPath rules it has no business carrying.

So I stopped borrowing. The filter grammar moved into padvinder as its own recursive-descent parser written against the RFC, and the xprsn dependency came out with it. That is the real reason padvinder has zero runtime dependencies: the one piece it could have reused is the one piece the standard needed it to own.

Why it can't be turned into code

The safety is the same compile-to-closures technique I wrote up for xprsn, so here is only its shape. Every selector and every filter becomes a small function that already exists in the shipped source, and a parser wires them together. Query text never gets reassembled into JavaScript, which leaves nothing to inject into and nothing for a CSP to block.

The test suite keeps that honest by running under node --disallow-code-generation-from-strings, the flag that turns every eval-shaped call into a thrown error, exactly as a strict CSP does in the browser.

The prototype escapes are closed as well. __proto__, constructor, and prototype never match, in a path or a filter, so [?@.constructor] finds nothing rather than climbing toward Function. Matching reads own properties only, and a query never writes back to the data you gave it. A test snapshots the input and compares it afterward to prove that last one.

Is it the right tool?

If your paths are all written by you, committed to your repo, and never near a strict CSP, you do not need any of this, and a bigger library with a wider feature set is a reasonable pick. padvinder is built for the narrower spot: the query might come from a user or a config row, the page runs under a CSP you will not weaken, and you would rather the results match the standard than one project's reading of it. The 456-case suite is what I lean on there. It is not my definition of correct, it is the RFC's.

Getting started

npm install padvinder

The README has the full selector table, the filter grammar, and the safety notes. Use find when you want an answer now and query when you will ask the same question a lot.

Closing thoughts

The satisfying part of this one was not the size or even the safety. It was having a compliance suite play judge. Most of my small libraries are correct by my own definition, which is a comfortable and slightly circular place to sit. Here, 456 cases I did not write got a vote, and "I think this is right" became "the standard says this is right." That judge is also what ended the borrowing: you do not approximate your way through 456 cases, so the evaluator I had hoped to reuse had to go. I would take the trade on more projects if the specs existed for them.

If you have been running JSONPath over input you did not write yourself, I would like to hear how you drew the line around it.

Working with nested JSON: exploring it, mocking it, and querying it in Postgres

Fri, 03 Jul 2026 13:52:48 +0000

Flat JSON is easy. Nested JSON is where the day goes: an API response with orders inside customers inside accounts, a jsonb column three levels deep, a config file that grew a tree. Three tasks come up over and over, and each one has a trick that makes it painless. Here they are.

1. Reading a huge nested payload

Pretty-printing a 4,000-line response does not actually help. The indentation is correct, but finding one thing still means scrolling through thousands of lines. The fix is to stop treating JSON as text to scroll and start treating it as data to query.

Say the question is: does any item in this order have a null price?

{
  "account": {
    "id": 91,
    "customer": {
      "orders": [
        { "id": 4821, "items": [ { "sku": "A-19", "price": 12.5 } ] },
        { "id": 4822, "items": [ { "sku": "B-04", "price": null } ] }
      ]
    }
  }
}

Three moves turn this from a scroll into a lookup:

Filter by path, key, type, or value. Filtering to price pulls every price field out of the whole tree at once. Filtering by the value null jumps straight to the offending row. The needle in 4,000 lines becomes a single filtered result.
Select the nesting level. Instead of expanding everything, expand the tree to level 2 to see the shape of the data, then drill only into the branch that matters. This is the difference between reading structure and drowning in leaves.
Switch to a table view when a branch is really a list of records. Two hundred objects with the same keys is a spreadsheet pretending to be JSON, and reading it as rows and columns is far faster than reading it as nested braces.

Once JSON is filterable and foldable, payload size stops mattering.

2. Generating realistic nested JSON, fast

The opposite problem: there is no payload yet, and building a parser, a UI, or a test needs one. Hand-writing mock data is slow, and worse, it is too clean. Every record ends up with the same id, the same date, the same shape, which is exactly the uniformity that hides off-by-one bugs, dedupe bugs, and sort bugs.

Two shortcuts cover most cases:

Load a realistic nested sample. Start from a ready-made nested document (an order, a log line, an analytics event) and adapt it, instead of inventing structure from scratch.
Multiply one object into many varied records. Write a single object by hand, pick a count, and get back an array where fields like id, email, and dates actually vary per record. That variation is the point: a list of 50 rows that are genuinely different is what surfaces the bug that 50 identical rows would hide.

Realistic nested test data in seconds beats a hand-typed array of clones every time.

3. Querying and updating nested JSON in Postgres

Eventually that nested JSON lands in a jsonb column, and a viewer cannot help anymore. This is where people get stuck, so here is the practical core.

Assume a table orders with a jsonb column data holding the payload above.

Selecting. The two operators to internalize are -> and ->>. -> returns JSON, ->> returns text:

-- returns jsonb
SELECT data -> 'account' -> 'customer' FROM orders;

-- returns text (note the final ->>)
SELECT data -> 'account' ->> 'id' AS account_id FROM orders;

For a deep path, #> and #>> take an array of keys and are much cleaner than chaining arrows:

-- text value deep in the structure
SELECT data #>> '{account,customer,orders,0,id}' AS first_order_id
FROM orders;

To filter rows by a nested value, containment (@>) is both readable and indexable with a GIN index:

SELECT * FROM orders
WHERE data @> '{"account": {"id": 91}}';

And to work across an array, expand it into rows with jsonb_array_elements:

SELECT elem ->> 'id' AS order_id
FROM orders,
     jsonb_array_elements(data #> '{account,customer,orders}') AS elem;

Updating. The workhorse is jsonb_set(target, path, new_value). The gotcha that trips everyone: new_value must itself be jsonb, so cast it.

-- set a deep scalar (note the ::jsonb cast)
UPDATE orders
SET data = jsonb_set(data, '{account,customer,orders,0,id}', '9001'::jsonb)
WHERE id = 1;

Add or replace a key at a level with the || merge operator, and remove a path with #-:

-- shallow-merge a new key into the top level
UPDATE orders SET data = data || '{"archived": true}'::jsonb WHERE id = 1;

-- delete a nested key
UPDATE orders SET data = data #- '{account,customer,orders,0,items}' WHERE id = 1;

One honest warning: || merges only one level deep, so it will not deep-merge nested objects. For nested changes, reach for jsonb_set. And the genuinely hard case, updating a specific array element chosen by a condition rather than a fixed index, needs more than jsonb_set alone. That is a whole topic on its own, and it is the one worth reading a dedicated write-up on before you fight it in production.

Here is the honest reason all of this matters: nobody keeps these operators in their head. It is -> versus ->>, the #>> path syntax, the ::jsonb cast that jsonb_set quietly requires, the fact that || only merges one level. I forget them between projects every single time. So I wrote them down as a permanent, on-page reference next to the formatter: Querying nested JSON in PostgreSQL (SELECT and UPDATE) collects the common reads and writes in one place, for the moment memory does not serve. It is reference for the everyday cases, not a full Postgres manual.

Where the tool fits

I built all three of these into one place, at catssaymeow.org/json-formatter, because I got tired of one tab that only pretty-prints and another that only explores.

It formats, validates, and minifies (invalid JSON is pinpointed by line and column, not a bare "invalid"), and then lets you actually work with the structure: explore it as a tree, graph, or table, filter by path, key, type, or value, and expand to a chosen nesting level. When there is no data to work with yet, it can load a realistic nested sample or multiply one object into an array of varied mock records. It runs as client-side JavaScript, so the JSON stays in the browser, and clean JSON can be sent straight to a built-in API Client as a request body.

And for the part that happens after the database call, the same page carries a Postgres jsonb querying reference for the SELECT and UPDATE syntax that never sticks. Three recurring nested-JSON tasks, one page to come back to.

If you put it up against your own worst nested payload, I would like to hear what it struggles with.

How to View and Analyze Large JSON Files Without Getting Lost

Tue, 09 Jun 2026 19:00:32 +0000

You open a JSON file. Your text editor freezes for three seconds. When it finally loads, you're staring at 80,000 lines of tightly packed, unindented data — no line breaks, no color, no structure. You need to find one specific value buried somewhere in a deeply nested object. You scroll. You squint. You ctrl+F for a key name and get 47 matches.

Sound familiar?

Working with large JSON files is one of those quietly painful parts of being a developer, data analyst, or QA engineer. JSON is everywhere — API responses, config files, exported datasets, logs, database snapshots — and while the format itself is elegantly simple, it scales poorly to human readability. A 200-line JSON file is manageable. A 20,000-line file is a different beast entirely.

This guide walks you through the practical strategies, tools, and techniques that make it possible to inspect, navigate, and analyze JSON data efficiently — without losing your mind in the process.

Why Large JSON Files Are So Hard to Read

JSON's design is optimized for machines, not humans. It's a lightweight data interchange format, which means it prioritizes parsability over readability. When you're dealing with small payloads, this doesn't matter much. But as files grow, a few structural realities make things difficult:

Deeply nested objects. Real-world JSON structures often have five, six, or even ten levels of nesting. An e-commerce API response, for example, might have an order object that contains a customer object, which contains an address object, which contains a coordinates object — and you need to reach order.customer.address.coordinates.lat. In raw text, tracing that path is genuinely difficult.

Minification. Many APIs and systems return minified JSON — everything on a single line, all whitespace stripped. This is efficient for transmission but completely unreadable without formatting.

Lack of visual hierarchy. Without indentation and color coding, objects and arrays blur together. You lose track of where one object ends and another begins.

Scale. A paginated API response with 500 records, each containing 30 fields, produces 15,000 key-value pairs. A database export can easily run into millions of lines. Searching linearly through that volume is impractical.

Common Challenges When Working with Large JSON Datasets

Before diving into solutions, it helps to name the specific problems developers encounter most often:

Finding a specific key or value is the most common task, and it's deceptively hard when a key name like id or status appears hundreds of times across different nested levels.

Understanding the overall structure is another major challenge. Before you can work with a JSON file, you need to understand its shape — what top-level keys exist, what type each value is, how deep the nesting goes. In a large file, this isn't obvious at a glance.

Comparing records across a large array is tedious without a proper view. If you're reviewing 200 user objects and need to spot inconsistencies in their data structure, raw text makes this nearly impossible.

Validating data integrity — checking that required keys exist, values are the right type, arrays aren't unexpectedly empty — is a real task for QA engineers and data analysts that becomes error-prone without the right tooling.

Performance issues are practical too. Many text editors struggle to syntax-highlight or even open files larger than a few megabytes. Some will outright crash.

Getting Oriented: Understanding JSON Structure

Before you can analyze large JSON files, you need a solid mental model of JSON structure. Everything in JSON falls into one of these types:

Objects: Curly braces {}, containing key-value pairs
Arrays: Square brackets [], containing ordered lists of values
Strings: Quoted text like "hello"
Numbers: Numeric values like 42 or 3.14
Booleans: true or false
Null: The absence of a value

Here's a small example to illustrate a real-world nested structure:

{
  "order": {
    "id": "ORD-9921",
    "status": "shipped",
    "customer": {
      "name": "Priya Sharma",
      "email": "*Emails are not allowed*"
    },
    "items": [
      {
        "product_id": "P-441",
        "name": "Wireless Keyboard",
        "quantity": 1,
        "price": 49.99
      },
      {
        "product_id": "P-882",
        "name": "USB Hub",
        "quantity": 2,
        "price": 19.99
      }
    ],
    "total": 89.97
  }
}

This is manageable at 25 lines. Now imagine 500 orders in an array, each with a deeper customer profile, multiple shipping addresses, discount codes, tax breakdowns, and fulfillment metadata. That's when JSON file analysis becomes a serious challenge.

The Power of JSON Tree View

The single biggest improvement you can make when working with large JSON files is switching from a flat text view to a JSON tree view.

A tree view presents the JSON structure hierarchically, like a file explorer. Top-level keys are shown as expandable nodes. You click a node to expand it and see its children. Arrays show item counts. Objects show their keys. You can collapse an entire branch when you're not interested in it, and expand only the paths you care about.

This changes the experience fundamentally. Instead of reading 10,000 lines of text sequentially, you navigate a structure. You instantly see that the top-level object has four keys. You expand data, see it's an array of 200 items. You expand the first item to understand the record schema. You check that the schema looks consistent across a few more items. You've oriented yourself in thirty seconds instead of five minutes.

JSON Viewer is a tool that does exactly this. Paste your JSON or upload a file, and it renders the structure as an interactive tree. Nodes expand and collapse on click. You can see at a glance whether a value is a string, number, array, or nested object. For anyone regularly dealing with complex API responses or large configuration files, this kind of visualization is not a luxury — it's a practical necessity.

Techniques for Navigating Nested JSON Structures

Once you have a tree view available, there are specific techniques that make navigation faster and more reliable.

Start from the top level. Before drilling into any specific part of the data, expand only the top-level keys to understand the overall shape of the file. Is the root an object or an array? How many top-level keys are there? What are their names? This gives you a map before you start exploring.

Use expand/collapse aggressively. When inspecting large JSON data, treat collapsed nodes as closed drawers. Only open the ones you need. This prevents cognitive overload and keeps your focus on the relevant portion of the data.

Follow the path, don't scroll. When you need to reach a specific nested value, traverse the tree path deliberately: expand the parent key, then the child, then the next level. This is far faster than scrolling through raw text looking for the matching closing brace.

Pay attention to types. A tree viewer makes it easy to spot when a value that should be a string is actually a number, or when a field that should always be an array is sometimes null. These type inconsistencies are common bugs in API integrations and are very easy to miss in raw text.

How to Search Within Large JSON Files

Searching is where many developers waste the most time. Using a plain text search (ctrl+F) in a JSON file is blunt: it finds all occurrences of a string, regardless of where they appear in the structure. If you're searching for "status" in an API response that has nested objects for orders, line items, shipping, and payment — you'll get matches on every level.

Smarter approaches:

Use JSON path notation mentally. Before searching, identify the path you're interested in. For example, you're not looking for status anywhere — you're looking for order.payment.status. This narrows your target significantly.

Use jq for command-line filtering. jq is a lightweight command-line tool for querying JSON. It's powerful and precise:

# Extract all order IDs from a large orders array
cat orders.json | jq '.orders[].id'

# Find all items where status is "failed"
cat response.json | jq '.[] | select(.status == "failed")'

# Get a specific nested value
cat data.json | jq '.results[0].customer.email'

jq is especially valuable when dealing with large JSON files in automated pipelines or when you need to extract specific data without loading the entire file into an editor.

Use Python for structured inspection. For one-off analysis tasks, Python's json module lets you load and query JSON data with full programming logic:

import json

with open('large_data.json') as f:
    data = json.load(f)

# Check the top-level structure
print(type(data))
print(data.keys() if isinstance(data, dict) else f"Array with {len(data)} items")

# Find all unique status values
statuses = set(item['status'] for item in data['orders'])
print(statuses)

This approach is ideal when you need to understand patterns across a large dataset, not just find a single value.

Analyzing API Responses That Span Thousands of Lines

API responses are one of the most common sources of large JSON files. When you're debugging an integration, reviewing a webhook payload, or testing a third-party API, the response might contain thousands of lines with nested pagination metadata, embedded related objects, and arrays of arrays.

A few practical strategies:

Inspect the envelope first. Most APIs wrap their data in a standard envelope: data, results, items, or records as the main payload, alongside meta, pagination, errors, or status at the top level. Understanding the envelope structure tells you where the actual data lives and where the metadata lives.

Sample your records. When the response contains a large array of records, you don't need to look at all of them to understand the schema. Inspect three to five items in detail — the first, one from the middle, and the last. Look for any structural differences. If the schema is inconsistent across records, that's a bug worth flagging.

Check for nulls and empty arrays. Large API responses often contain optional fields that are populated inconsistently. A field that's an object in one record might be null in another. A tree view makes these stand out visually.

Compare actual vs. expected structure. If you have API documentation, use it alongside a tree view of the actual response. Discrepancies — missing fields, wrong types, unexpected nesting — are much easier to spot when you can see the full structure laid out visually.

Reviewing Config Files and Exported Datasets

Configuration files in JSON format — think package.json, .prettierrc, AWS CloudFormation templates, or Kubernetes config files — have their own inspection challenges. They're typically not large by record count, but they're structurally complex, with nested objects for different environments, feature flags, permissions, and resource definitions.

When reviewing these:

Understand the schema before editing. Modifying a deeply nested config without understanding its parent structure is a common source of bugs. Use a tree view to confirm the full path to the value you're changing before touching anything.

Validate before deploying. Many config formats have strict schema requirements. A JSON viewer that highlights syntax errors saves significant debugging time.

Exported datasets from databases or analytics tools present a different challenge: they're often large flat arrays with many fields per record. For these, tools like jq or Python's pandas library are more appropriate for analysis than a tree viewer, since you're doing aggregate analysis rather than structural inspection.

Common Mistakes to Avoid

Even experienced developers make these mistakes when dealing with large JSON files:

Opening raw JSON in a basic text editor. Notepad, nano, or even VS Code without a JSON extension will show you a wall of text. Always use a viewer or formatter as your first step.

Searching too broadly. Searching for a key name without considering where in the structure it appears leads to confusing results. Be specific about the path you're looking for.

Ignoring schema inconsistencies. When a field is missing from some records or has an unexpected type, it's easy to dismiss it as a fluke. It rarely is. Schema inconsistencies are almost always bugs.

Editing large JSON files manually without validation. Adding a comma in the wrong place, forgetting a closing bracket, or mismatching braces will silently break your file. Always validate after editing.

Assuming minified JSON is corrupted. Minified JSON looks like garbage but is valid. Run it through a formatter before assuming something is wrong.

Practical Tips That Save Real Time

To close out, here are the habits that make a measurable difference when you work with JSON data regularly:

Always format before you read. Paste minified JSON into a formatter or viewer as your first step, every time.

Use a tree view for structure, jq for data extraction. These two tools complement each other: the tree view tells you the shape, jq lets you query the content.

Keep a scratch file. When analyzing a large file, paste the specific subtree you're working with into a scratch file. You don't need to keep the whole 10,000-line file open to work on one nested object.

Note the array lengths. When a field is an array, check its length. An empty array where you expect data is a very common source of bugs that's easy to overlook in raw text but obvious in a tree view.

Validate early and often. If you're writing or modifying JSON, validate it before it goes anywhere. A quick paste into a JSON viewer catches most structural errors immediately.

Conclusion

Working with large JSON files doesn't have to be a frustrating experience. The core problem — that JSON scales poorly to human readability — is solvable with the right tools and techniques. A tree view transforms an opaque wall of text into a navigable structure. Command-line tools like jq let you query precisely instead of searching blindly. Python gives you full analytical power when you need to understand patterns across thousands of records.

The key takeaways: always format before you read, use tree views to understand structure, search with path awareness instead of plain text search, and validate often. Whether you're debugging an API integration, reviewing a configuration file, or analyzing an exported dataset, these habits will consistently save you time and prevent the kind of errors that only show up in production.

JSON is a tool, and like any tool, it rewards the people who learn to use it properly.

Why Your React Frontend Crashes When an LLM Streams Malformed JSON

Tue, 09 Jun 2026 09:54:30 +0000

A production walkthrough with a live Next.js demo - JSON.parse() vs partial-json + Zod for real-time AI dashboards.

We don't prompt AI, we architect it.

Article content
If you ship real-time AI dashboards in React, you have probably seen this pattern:

Stream starts.
Metrics update for a few chunks.
Console throws SyntaxError.
UI goes blank.

The bug is usually not React. It is not Next.js.

It is calling JSON.parse() on a live stream buffer an unterminated string that is incomplete by design.

I built an open-source demo to make this failure obvious - and to show a resilient parsing pipeline you can copy into production.

Most tutorials teach you to crash. Production demands resilience.

Tutorial path:

→ JSON.parse(buffer) on every chunk

→ UI crashes on a single missing bracket

Production path:

→ partial-json + Zod .partial()

→ UI updates live, ignoring malformed chunks

Error boundaries: seatbelt, not engine

StreamErrorBoundary catches render errors in children. It does not replace safe streaming parse logic or catch async errors inside useEffect.

Use error boundaries as a safety net after you fix parsing strategy.

Run it locally in 5 minutes

git clone https://github.com/gauravthorath/react-llm-stream-json-demo.git

cd react-llm-stream-json-demo

pnpm install

pnpm run dev
Open http://localhost:3000 → click Run Stream → open DevTools on the left panel.

Architecture notes: https://github.com/gauravthorath/react-llm-stream-json-demo/blob/main/ARCHITECTURE.md

Production takeaway

Parse partial: do not JSON.parse the live buffer
Validate partial: Zod .partial().safeParse
Render partial: skeletons for missing fields
Retain last good snapshot: do not blank UI on trailing garbage

Parse partial. Validate partial. Render partial.

What is next

Follow-up: tool-calling streams and schema drift - when the model changes field shapes mid-session and your UI needs a migration strategy, not just a parser.

Comment Part 2 on the video if you want that teardown next.

Connect

YouTube: https://youtu.be/n51RAGJZGdQ

GitHub: https://github.com/gauravthorath/react-llm-stream-json-demo

Portfolio: https://gauravthorat-portfolio.vercel.app/blog/react-llm-stream-json-parser

react #nextjs #typescript #ai #frontend #webdevelopment #javascript #softwareengineering #llm #sse

fastjson 0.3.0: A Faster Drop-In ext/json for PHP, Backed by yyjson

Wed, 20 May 2026 11:49:28 +0000

// Before
$payload = json_encode($response);
$data    = json_decode($input, true);
$ok      = json_validate($input);

// After
$payload = fastjson_encode($response);
$data    = fastjson_decode($input, true);
$ok      = fastjson_validate($input);

That's the migration. Search-and-replace json_* for fastjson_*. JSON flags, error constants, and last-error semantics carry across byte-for-byte. The two extensions sit next to each other in the same process; adoption is per call site, not per repo.

On the simdjson_php canonical 14.8 MB corpus, that swap buys 6.06× encode throughput, 2.66× decode, and 5.10× validate against ext/json on the same PHP 8.6.0-dev build. The repo is at github.com/iliaal/fastjson. 0.3.0 shipped yesterday.

Why drop in faster JSON for PHP

ext/json is fine. It's correct, well-maintained, and tracks the spec. On low-traffic endpoints it isn't on anyone's profiler. The cost shows up at scale: any application that calls json_encode and json_decode on every request path eventually finds JSON serialization sitting at the top of a flame graph. API gateways feel it first, then log processors and microservice fan-out paths.

Before fastjson the practical options were two:

Stay on ext/json. Eat the CPU cost.
Reach for simdjson_php. It's fast but decode-only and not API-compatible; every call site has to be rewritten around its result shape.

fastjson is option three. It's a native PHP extension that wraps yyjson 0.12.0 (MIT, ~6K LOC of focused C) behind a namespaced API that mirrors ext/json's contract. PHP 8.3 minimum; 8.4 and 8.5 supported; coexists with ext/json.

What "drop-in" actually means here

The risk with any "drop-in" claim is that it covers 90% of cases and silently changes behavior on the 10% that matter. So this section is what fastjson does and what it doesn't:

Function signatures track ext/json. fastjson_encode($value, int $flags = 0, int $depth = 512). Same positional shape. Same defaults. Same return values on success, false on failure.
*`JSON_ flags match byte-for-byte.** JSON_UNESCAPED_SLASHES, JSON_UNESCAPED_UNICODE, JSON_PRETTY_PRINT, the JSONHEX* family, JSON_THROW_ON_ERROR, JSON_INVALID_UTF8_IGNORE, JSON_INVALID_UTF8_SUBSTITUTE`. The integer constants are intentionally identical so user code can pass the same flag value into either function.
*`JSONERROR codes match byte-for-byte, messages don't.** fastjson_last_error() returns the same JSONERROR* int as json_last_error() for the same failure class, so code branching on error codes works without modification. fastjson_last_error_msg() returns yyjson's parser message (e.g., "unexpected character"`), not ext/json's. Application code that pattern-matches on the message string needs updating; code that branches on the code does not.
Coexistence, not replacement. Both extensions load. Migrate the call sites where JSON is on the hot path; leave the rest on ext/json.
*53 phpt tests rewritten from `php-src/ext/json/tests/.phpt** run alongside fastjson's own suite. The rest of the upstream suite is categorized in tests/upstream-json/.skiplist with the reason each test was deferred (most are tests of ext/json` internals that don't translate, a few hit known divergences).
Documented divergences. Large/scientific doubles emit 100000000000000000.0 where ext/json emits 1.0e+17 in some ranges. U+2028 / U+2029 line separators emit as ordinary code points; ext/json always escapes them for JSONP safety. Both divergences are in the skiplist with rationale.

The numbers

Full simdjson_php canonical corpus: 14.8 MB across 15 files, the same set the simdjson PHP binding has been benchmarked against for years. Hardware: i9-13950HX. PHP 8.6.0-dev, release build (-O2). fastjson built -O2 against the same PHP. yyjson 0.12.0 with three local patches. Numbers in throughput, MB/sec:

Operation	fastjson	ext/json	Speedup
Decode (stdClass)	602 MB/s	227 MB/s	2.66×
Decode (assoc array)	628 MB/s	237 MB/s	2.65×
Encode	1,092 MB/s	180 MB/s	6.06×
Validate	1,352 MB/s	265 MB/s	5.10×

Visual side-by-side, including ext/json with Nora Dossche's open SIMD encode work (php-src#17734) and simdjson_php on the same PHP 8.6.0-dev build, lives at https://iliaal.github.io/fastjson/baseline.html. Reproduce locally with bench/run.php against any PHP install.

The encode speedup is the largest gap because the PHP-native encoder has the most room to give back. ndossche's php-src#17734 patch closes a meaningful chunk of that gap inside ext/json itself using SIMD on string encoding. fastjson and that PR attack the same problem from different angles, and an application can benefit from both once #17734 lands upstream (fastjson re-baselines automatically; the visual page already shows both).

How the encoder gets to 6×

The encoder is one-stage. A zval walks straight into a smart_str buffer via yyjson's writer primitives. There's no intermediate DOM, no two-pass build-then-serialize, no temporary string allocations for the common-case scalars. Each PHP value type maps to one or two yyjson calls; arrays and objects walk recursively into the same path.

A few less-obvious pieces:

Custom allocator wired through Zend. yyjson's allocator hooks route every alloc/realloc/free through PHP's emalloc family. JSON workloads count against memory_limit and against per-request memory accounting; oversized inputs bail out the same way any other PHP allocation does.
Direct smart_str integration. No RETURN_STRING(estrdup(buf)) after a separate allocation. The yyjson writer writes into the smart_str.s backing store, which becomes the return zend_string directly. One allocation per encode call in the common case.
*`HEX_ flag scan-first.** The flags JSON_HEX_TAG/HEX_AMP/HEX_APOS/HEXQUOT rewrite specific characters into hex escapes. fastjson scans the string once for any candidate character; if none are present, the rewrite path is skipped entirely and the string is encoded directly. Defensive callers that pass HEX*` flags as a precaution on payloads that don't actually contain the substituted characters don't pay the rewrite cost.
Integer-valued-double shortcut. When a double round-trips losslessly through int64, fastjson emits it as an integer without going through php_gcvt or yyjson's REAL writer. A cheap range check fires before floor(), so number-heavy arrays of non-integer doubles don't pay libm per element.

The decode path's gain has different sources: yyjson itself, which parses with less branching and tighter memory locality than ext/json's parser, and a local yyjson patch (YYJSON_READ_VALIDATE_ONLY) that turns the read path into a fast validate-only mode without materializing values.

The memory tradeoff

Worth surfacing before anyone hits it in production. Decode and validate hold the yyjson document object in memory alongside the PHP-side result, because yyjson's value graph is built first, then traversed to produce zvals. Peak heap on decode is roughly 1.7× what ext/json peaks at on the same input. Encode is one-stage and peaks at ~1.06× of ext/json.

Validate is the loudest: peak heap is ~101× ext/json's streaming validator (which sits at a constant ~80 bytes since it doesn't materialize anything). The headline number sounds extreme, but the absolute footprint is bounded by yyjson's read path, and it's already 2.7× lower than yyjson's stock read path thanks to a vendored validate-only patch (YYJSON_READ_VALIDATE_ONLY) that skips the value-graph build.

For most callers, the speedup wins. If the application is validate-heavy on giant inputs under tight memory_limit, the memory profile is a real consideration. The right move there is to leave validate-on-huge-inputs on ext/json and migrate the encode and decode paths. That's exactly what coexistence buys.

Compat harness

53 rewritten phpt tests from php-src/ext/json/tests/*.phpt run alongside fastjson's native suite. They cover the common decode/encode/validate paths, the flag combinations, and the documented error conditions.

The two remaining intentional divergences:

Large/scientific doubles. Outside the integer-valued-double shortcut range, fastjson emits yyjson's real-number format. ext/json uses php_gcvt and switches to scientific notation earlier. The disagreement window narrowed in 0.3.0; what's left is the genuinely-fractional case where yyjson and php_gcvt produce different decimal representations of the same IEEE 754 double.
U+2028 / U+2029 line separators. ext/json always escapes these for JSONP safety. yyjson treats them as ordinary code points. fastjson follows yyjson's behavior. If JSONP is in the deployment path, set JSON_UNESCAPED_UNICODE off in ext/json and stay on ext/json for that endpoint, or wrap fastjson output through a JSONP-safe post-step.

Upstream collaboration with yyjson

fastjson vendors yyjson 0.12.0 with three local patches. Full details are in vendor/yyjson/PATCHES.md; the short version of each, and what happened when each was proposed upstream:

Lowercase hex digits in \uXXXX escape table. yyjson emits uppercase; ext/json emits lowercase. RFC 8259 §7 allows either, but byte-parity with ext/json is the project's compat goal. Proposed as yyjson#264 (YYJSON_WRITE_LOWERCASE_HEX flag) and accepted upstream on 2026-05-11. fastjson will drop this patch once the vendored sources advance past yyjson 0.12.0.
YYJSON_READ_VALIDATE_ONLY, no-tree validate mode. Forks yyjson's parser entry point to skip the value-graph build entirely; peak memory drops 2.7× on the validate corpus. Not yet proposed upstream; the API surface needs a round of review before it's ready to submit.
Public yyjson_write_string_to_buf() wrapper. Exposes yyjson's internal direct-write primitive so fastjson's one-stage encoder can compose at the buffer level. Proposed as yyjson#266 and closed upstream; the maintainer preferred to keep that surface private. The wrapper stays vendored locally; fastjson lives with it indefinitely.

Install

pie install iliaal/fastjson

Or build from source:

phpize
./configure
make -j
make test
sudo make install

PHP 8.3 minimum. No external library dependencies; yyjson is vendored in src/yyjson/.

The four-character migration

fastjson stays useful as long as yyjson's design choices (one-stage encoder, validate-only fast path, allocator hooks) beat what fits into ext/json's compatibility envelope. For the call sites where JSON serialization shows up on a flame graph, the migration is four characters; the rest of the codebase doesn't have to care.

Repo: https://github.com/iliaal/fastjson. Benchmark methodology and reproduction: bench/run.php. Visual baseline: https://iliaal.github.io/fastjson/baseline.html.

Parsing hundreds of megabytes of JSON in milliseconds in React Native

Tue, 19 May 2026 08:39:22 +0000

How JSON.parse materializes entire documents in JavaScript, and how react-native-fast-json uses native simdjson + lazy JsonView access to change the cost model in React Native.

react-native-fast-json is an open-source React Native library that wraps simdjson in C++ and exposes a lazy JsonView through Nitro Modules.

Most teams treat JSON parsing as “cheap” because API responses are small. The mental model is:

bytes on wire → JSON.parse → plain objects in JS → app logic

That model breaks when the payload is hundreds of megabytes. The dominant cost is not “JSON syntax is slow” in the abstract, it is allocating and wiring up a full value tree in the JavaScript heap.

What `JSON.parse` actually does in React Native

When you call JSON.parse (or await response.json()), the engine:

Holds the UTF-16/UTF-8 string (or an internal representation of the body)
Walks the entire document
For every object, array, number, boolean, and string, creates JavaScript values with engine overhead (hidden classes, property storage, array backing stores, interned strings, etc.)

For a ~250 MB file, outcomes depend heavily on platform and available RAM:

Symptom	Typical cause
Multi-second freezes	Single-threaded parse + massive allocation on the JS thread
Memory far above file size	JS object graph overhead (often ~2–4× the raw JSON size in bad cases)
OOM / process kill	Peak JS heap + retained tree + rest of the app

On the data_250mb.json fixture in our tests: iOS could complete JSON.parse but only with extreme CPU and memory spikes; Android always crashed before parse finished, making JSON.parse impossible for that payload on Android, while native parseFile worked on both platforms.

Benchmark fixture (source)

Measurements in this repo use the public ~250 MB JSON from antonmedv/json-examples:


Repository	https://github.com/antonmedv/json-examples
File	`data_250mb.json`
Raw URL	`https://github.com/antonmedv/json-examples/raw/refs/heads/master/data_250mb.json`

The example app downloads that URL, caches it on disk, then calls parseFile on the local path.

Measured results (`parseFile`, release builds, physical devices)

Platform	Time to root `JsonView`	CPU / memory (observed)
iOS	~100 ms	Stable, no long CPU peg, no runaway JS heap growth during parse
Android	~500 ms	Stable, same qualitative behavior as iOS, slower wall time

JSON.parse / response.json() on the same fixture:

Platform	`JSON.parse`	`parseFile` + lazy `JsonView`
iOS	Many seconds; extreme CPU and memory spikes (~1.2 GB JS heap ballpark)	~100 ms; CPU and memory stable
Android	Crash, could not complete parse	~500 ms; CPU and memory stable (~400 MB mostly native buffer)

The second row is not “free memory.” The file still occupies roughly file size + simdjson padding in native memory. The win is avoiding a second full copy as a giant JS object graph until you explicitly materialize pieces.

Why “read file as string, then `JSON.parse`” can be worse at peak

A common variant:

const text = await readFile(path, 'utf8');
const data = JSON.parse(text);

At peak you may briefly hold:

The entire file as one JS string
The entire parsed tree
Parser scratch allocations

response.json() can sometimes avoid exposing one giant application-level string, but the end state is still the same: a full tree in JS. For large files, peak heap is the killer.

The technique: how react-native-fast-json does it

react-native-fast-json is what makes the native lazy model practical in React Native. Instead of returning a plain object from JSON.parse, it shifts work across a different boundary:

file bytes → native buffer (simdjson) → JsonView handle → read paths on demand

1. Native buffer (simdjson)

parseFile loads the file into a padded native buffer (simdjson::padded_string::load). Parsing runs in C++ with simdjson (SIMD-accelerated, widely used for large JSON).

The document’s canonical bytes live in native memory, not as nested JS objects.

2. Lazy `JsonView` (Nitro hybrid object)

The root returned to JavaScript is a JsonView, a Nitro hybrid object backed by C++. Calling:

getValue('metadata')
atPath('$.a.b.c')
asString() on a scalar

…does targeted native work and only crosses the bridge with small results (another JsonView handle or a primitive), not the whole document.

3. Materialization is explicit and expensive

When you call:

asObject(), subtree becomes a JS-friendly map/array structure
rawJson(), subtree becomes a JS string

…you have chosen to pay the JS heap cost for that slice. On a 250 MB document, calling these on the root is equivalent to opting back into the problem you avoided.

Architecture

HybridFastJson (cpp/HybridFastJson.cpp):

parseFile(path), load file, construct HybridJsonView, cache by path string
parseString(str), copy string into native view (not path-cached)
release(path), erase cache entry for that path

HybridJsonView, navigation, path helpers, scalar coercion, optional materialization.

Until release(path), a cached parseFile keeps the full native buffer alive even if JS drops its JsonView reference, plan releases when the flow ends.

What you still pay for (honest accounting)

Cost	Still there?
File-sized native memory	Yes, roughly JSON size + padding
Parse CPU	Yes, but native simdjson is much faster than building a JS tree
JS heap for the whole tree	No, unless you `asObject()` / `rawJson()` large parts
Bridge traffic	Per access; keep reads coarse-grained for huge docs
Path cache	`parseFile` retains buffer until `release(path)`

There is no zero memory overhead. The claim that holds up is: near-zero additional JS heap for the full document while you navigate lazily.

Why milliseconds (or low hundreds of ms) are plausible for hundreds of MB

simdjson is designed for throughput on large valid JSON. On the data_250mb.json fixture, parseFile after the file is on disk landed at ~100 ms on iOS and ~500 ms on Android in our release builds, with stable CPU and memory. By contrast, JSON.parse on iOS took many seconds with extreme spikes; on Android it always crashed, so native parse was the only viable option there.

Caveats when you reproduce benchmarks:

Fixture, use the same public file or disclose yours
Cold vs warm storage (first read after download vs re-parse)
Platform, Android was ~5× slower than iOS in our tests; do not assume one number for all devices
Device tier and storage speed
JSON shape (depth, string density) affects constant factors
Debug vs release native builds, ship measurements on release binaries

Comparison table (same fixture, measured + observed)

Dimension	`JSON.parse` (our tests)	Native `JsonView` (`parseFile`)
iOS	Completes after many seconds; extreme CPU/memory spikes	~100 ms; stable CPU/memory
Android	Crash, parse not possible on ~250 MB	~500 ms; stable CPU/memory
Primary memory	Full JS object graph (when it survives)	Native padded buffer (~file size)
UI thread risk	High during parse	Lower; still avoid huge sync work in JS
Best access pattern	Whole-tree mutation	Few paths / scalars / wildcards
Foot-guns	Retaining whole `data`	`asObject()` / `rawJson()` on huge nodes; never calling `release`

Wildcards and paths

atPath('$.a.b.c'), simple dotted segments, no [index] in the path string.
*`atPathWithWildcard('$.items[].id')**, dynamic segments; returns string[] | null` for matched scalar strings.

For very large documents, prefer narrow paths over scanning wildcards across the whole tree unless you have measured the cost.

When native lazy parsing is the wrong tool

Not a common use case, prefer better alternatives

Shipping ~250 MB JSON to a phone is a smell, not a pattern. We benchmark it because it stress-tests the parser and matches real legacy pain, not because every app should do it.

Try these first (usually cheaper than optimizing parse):

Don’t send the file, paginate, stream, or return only fields the UI needs.
Don’t use JSON on the wire, schema-driven binary formats decode faster and smaller.
Don’t keep one blob, SQLite (or similar) with indexes for the queries you actually run.
Don’t parse on the critical path, generate summaries or shards on the server; sync deltas.

Native lazy parsing is a mitigation when you are locked into large JSON. It is not a substitute for a sane data contract.

When this library still does not help enough

Small payloads, JSON.parse is simpler and fast enough.
You need the entire document in JS anyway, you will materialize everything eventually; measure whether native parse + bulk asObject() is still a win.
Heavy in-place mutation of arbitrary subtrees in JS, work on plain objects.
You cannot hold file-sized native RAM (low-RAM devices + multiple huge files), budget and serialize access; even native buffers are ~file size.

Benchmark methodology

import { fastJson } from 'react-native-fast-json';

const root = await fastJson.parseFile('/path/to/large.json');
const version = root?.getValue('metadata')?.getValue('version')?.asString();
fastJson.release('/path/to/large.json');

Fixture, data_250mb.json from antonmedv/json-examples, via
https://github.com/antonmedv/json-examples/raw/refs/heads/master/data_250mb.json
RN Version 0.85.0, New Architecture.
Device, physical iOS and Android (document model, OS, RN version, Hermes on/off).
Build, release native binary.
Scenarios
- A: response.json() or JSON.parse after full string in memory
- B: download/cache to disk → parseFile(localPath) → read N fields → release(path)
Metrics, wall time to root JsonView, JS heap snapshot, CPU/memory stability in Instruments / Android Studio.
Our results, parseFile: iOS ~100 ms, Android ~500 ms, CPU/memory stable. JSON.parse: iOS many seconds with extreme spikes; Android always crashed on this fixture.

The example app implements download + parseFile + timing UI, use it as a starting point, not as universal truth.

Summary

Idea	Detail
Problem	`JSON.parse` materializes the whole document in the JS heap.
Technique	Parse in native with simdjson; expose `JsonView`; read lazily.
Speed	`data_250mb.json`: ~100 ms (iOS) / ~500 ms (Android) for `parseFile` in our tests.
CPU / memory	Stable on native parse; `JSON.parse` had extreme spikes on iOS and crashed on Android for this fixture.
Memory model	Avoid blowing up the JS heap; native still holds ~file size.
Discipline	`release(path)`, avoid huge `asObject()` / `rawJson()`, extract scalars to plain JS when done.

Try react-native-fast-json

yarn add react-native-fast-json react-native-nitro-modules

Install pods, rebuild, then use fastJson.parseFile for large files on disk. Peer dependency: react-native-nitro-modules.

Don’t Just Ask AI for JSON. Design the Output.

Wed, 13 May 2026 09:55:59 +0000

When we use AI in a chat window, a little extra text usually does not matter.

If the model says:

Sure, here is the result:

before answering, we can still read it.

If it adds:

Hope this helps!

after the answer, we can simply ignore it.

But code cannot ignore things the same way.

When AI output needs to be passed into a parser, stored in a database, sent to another service, or used inside an automation pipeline, the shape of the answer matters as much as the answer itself.

This is where many AI workflows break.

We ask:

Please output the result in JSON.

And the AI replies:

Of course! Here is the JSON:

```json
{
  "sentiment": "negative",
  "product": "Notebook Battery",
  "issue": "Battery lasts less than 3 hours"
}

Let me know if you need anything else!


To a person, this looks fine.

To code, this may be a problem.

The JSON may be wrapped in a code block.  
There may be text before it.  
There may be text after it.  
The keys may be wrong.  
The structure may change next time.  
The output may look like JSON but fail when parsed.

That is the structured output problem.

## “Give me JSON” is not enough

A common mistake is thinking that structured output is just a matter of asking more clearly.

```txt
Output in JSON.

Then:

Output only JSON.

Then:

Do not include any explanation. JSON only.

These instructions help.

But they do not fully solve the problem.

LLMs generate text probabilistically. They are very good at continuing patterns. In normal conversations, models often include polite openings, explanations, summaries, and helpful closing sentences.

So even when we ask for JSON, the model may still follow the conversational pattern it has learned.

This does not mean the model is broken.

It means we are asking a language model to produce something that is not really language in the usual sense. JSON is not a friendly paragraph. It is a strict structure.

And strict structures need more than polite instructions.

The three ways structured output often breaks

In practice, AI-generated JSON usually fails in a few familiar ways.

1. Text before the JSON

The model adds something like:

Sure, here is the result:

before the object.

A human can ignore it.
JSON.parse() cannot.

2. Text after the JSON

The model gives valid JSON, then adds an explanation:

This analysis shows that the user is unhappy with the battery performance.

Now the output is no longer pure JSON.

3. JSON-like text that is not valid JSON

This is the most subtle failure.

{
  sentiment: "negative",
  'product': "Notebook Battery",
  "issue": "Battery life",
}

It looks close enough to a human.

But it is not valid JSON.

Missing quotes, single quotes, trailing commas, comments, extra fields, wrong types — these are all small mistakes that can break a pipeline.

Structure is part of the design

The important shift is this:

Structured output is not just about formatting.

It is about designing the boundary between AI and the rest of the system.

When the AI is only talking to a person, the output can be flexible.

But when the AI is talking to code, the output must be predictable.

A paragraph is useful for reading.
A checklist is useful for review.
A table is useful for comparison.
JSON is useful when another system needs to process the result.

The same answer becomes more valuable when it has the right shape.

Start by separating thinking from output

One practical pattern is to separate the model’s reasoning space from the final output space.

Instead of saying:

Analyze this review and output JSON.

We can say:

First, analyze the review.

Then output only the final JSON in this exact format:

{
  "sentiment": "positive | neutral | negative",
  "product": "string or null",
  "main_issue": "string or null"
}

The idea is simple.

Let the model think freely where format does not matter.
Then make the final output strict.

This reduces the pressure on the model to reason and maintain a rigid format at the same time.

Examples are stronger than explanations

Another useful pattern is showing the model exactly what we want.

Instead of only explaining the format, provide examples.

Input:
"Galaxy S24 battery drains too fast."

Output:
{"sentiment":"negative","product":"Galaxy S24","issue":"battery life"}

Input:
"iPhone camera is amazing."

Output:
{"sentiment":"positive","product":"iPhone","issue":"camera"}

Now process this:

Input:
{{new_review}}

Output:

This works because the model continues the pattern.

If every example starts directly with JSON, the model is more likely to continue with JSON instead of adding a friendly sentence first.

In prompt engineering, examples often beat instructions.

For serious systems, prompts are not the final layer

Prompting can improve reliability.

But if the output is important enough, prompts alone may not be enough.

A production system may need stronger layers:

1. Clear output instructions
2. Separate reasoning from final output
3. Explicit schema
4. API-level structured output or tool calling
5. Validation and retry logic

This is the difference between asking for structure and designing for structure.

If the output fails, the system should not simply crash.

It should parse, validate, detect the error, and recover if possible.

For example:

The previous response could not be parsed as JSON.
The error was: trailing comma after final field.
Please return the same result as pure valid JSON only.

This kind of recovery loop treats AI output as something that can fail.

And that is a healthier assumption.

Structured output is about trust

The more AI becomes part of workflows, agents, and automation systems, the more important structured output becomes.

If AI is only helping us think, flexible text is fine.

But if AI output becomes input for another process, the format must be reliable.

A workflow cannot depend on “probably valid JSON.”
An automation cannot depend on “usually parseable output.”
A database pipeline cannot depend on “the model will likely follow instructions.”

At that point, structured output is not a nice-to-have.

It is part of system design.

The real lesson

The lesson is not:

Ask AI for JSON.

The lesson is:

Design the output so another system can trust it.

That means being clear about format, separating reasoning from final output, giving examples, using schemas when needed, and validating the result before trusting it.

AI can generate useful answers.

But useful answers are not always usable outputs.

When we work with AI as developers, we should not only ask:

Is the answer correct?

We should also ask:

Can this output be safely used by the next step?

That is the real reason structured output matters.

Originally published at Dechive:
https://dechive.dev/en/archive/prompt-structured-output

Unicode escape' codec can't decode bytes in position 2-3: truncated \ uxxxxxxxx escape

Sat, 09 May 2026 07:21:58 +0000

In this article, we will solve a syntax based problem of "Unicodeescape' codec can't decode bytes in position 2-3: truncated \uxxxxxxxx escape". This problem is encountered when we make a mistake in using the "unicode esacpe characters."Python's interpreter is designed to understand certain characters with certain symbols and in programming each character is given a unicode. This error occurs when the interpreter fails to distinguish between a normal character and a unicode character. We can solve this problem by simply manipulating the string according to the interpreter's logic. We will also discuss how these unicode character works and the reason behind this syntax problem. Let's quickly go through the contents of this article.

Table of contents: -

Understanding the error and discussing how the interpreter functions.
Solving the error with a relevant example
Why unicode characters were assigned and steps to avoid this error.
The Conclusion

Understanding the error and discussing how the interpreter functions. #

Let's break this error statment down: "Unicodeescape' codec can't decode bytes in position 2-3: truncated \uxxxxxxxx escape". This means that there is a syntax error and the interpreter is not able understand the meaning of this code. The "Unicodeescape" term means that the syntax error is caused because the unicode escape characters are not used properly. Now we have a lot to dissect, let's start with the interpreter's logic. if we pass a command like: -


print('This is her mother's dress')

The output of this code will be: -


print('This is her mother's dress.')
          ^^^^^^^^^^^^^^^^^^^^^
SyntaxError: invalid syntax. Perhaps you forgot a comma?

Here, the interpreter is unable to understand the range of the string. We use " ' " character to set the range of characters to be printed. The interpreter is confused because it is not able to distinguish between a normal character and a single quote range setter. We can solve this problem with the help of escape sequences.


print('This is her mother\'s dress')


This is her mother's dress

Here , the backslash character instructs the interpreter that it is an escape sequence for the single quote character ('). As we observed that the interpreter was confused and this confusion led to a syntax error. This is the exact problem we are facing. Let's understand this with the help of an example.

Solving the error with a relevant example#

We know that the interpreter follow certain rules while printing and traversing the data. Suppose we have to load a file and traverse the data stored in it. We use the file handling concepts and perform the operation but on running the program we encounter the same error: -


path = "C:\Users\Dell\Downloads\lamdaphagevirus\lamda.txt"
file = open(path, "r")
data=file.read()
print(data)


path = "C:\Users\Dell\Downloads\lamdaphagevirus\lamda.txt"
                                                              ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape

What went wrong here? The error is in the path variable. we passed the location of the file and it consists of multiple backslashes. Now the interpreter is confused which character is a normal character and which one is a unicode escape character. We know that the entire file location is a normal string without any unicode characters but the use of "\" characters is causing the problem. How are we going to solve this problem? Let's look at some solutions: -

Tip Each character is assigned with a unicode and an escape sequence which is universal for all the languages.

Solution 1

For the 1st solution, we will convert out string into a raw string like this: -


path = r"C:\Users\Dell\Downloads\lamdaphagevirus\lamda.txt"
file = open(path, "r")
data=file.read()
print(data)


GGGCGGCGACCTCGCGGGTTTTCGCTATTTATGAAAATTTTCCGGTTTAAGGCGTTTCCGTTCTTCTTCG
TCATAACTTAATGTTTTTATTTAAAATACCCTCTGAAAAGAAAGGAAACGACAGGTGCTGAAAGCGAGGC

A raw string is used whenever we want to avoid the technique of escaping characters with the help of backslashes. In order to convert a string into a raw string we have to simply add a "r" before the starting of the string. If this solution does not work for you then try the next method.

After this manually install "dlib" from this link. This can help the programmer to solve this issue.

Solution 2

If the the raw string method fails then try to escape the characters with the help of a backslash.


path = "C:\\Users\\Dell\\Downloads\\lamdaphagevirus\\lamda.txt"
file = open(path, "r")
data=file.read()
print(data)


GGGCGGCGACCTCGCGGGTTTTCGCTATTTATGAAAATTTTCCGGTTTAAGGCGTTTCCGTTCTTCTTCG
TCATAACTTAATGTTTTTATTTAAAATACCCTCTGAAAAGAAAGGAAACGACAGGTGCTGAAAGCGAGGC

For the interpreter a single backslash character means this "\\". All the escape characters begin with a backslash and by using this character before every backslash in the string, we can instruct the interpreter to treat this character as a escape sequence.

Solution 3

For the third solution we have to manipulate the string and replace every backslash with a "/" character.


path = "C:/Users/Dell/Downloads/lamdaphagevirus/lamda.txt"
file = open(path, "r")
data=file.read()
print(data)


GGGCGGCGACCTCGCGGGTTTTCGCTATTTATGAAAATTTTCCGGTTTAAGGCGTTTCCGTTCTTCTTCG
TCATAACTTAATGTTTTTATTTAAAATACCCTCTGAAAAGAAAGGAAACGACAGGTGCTGAAAGCGAGGC

Note In case of file handling, majority of the errors are caused due to an invalid location entry.

To know more about unicode errors vist this link.

Why unicode characters were assigned and how to avoid this error.#

Now that we have discussed the different solutions for the unicode error, we will try to understand the logic behind the designation of such codes. The main reason for assigning a universal code to every character was to establish uniformity in programming. During the early phases when technology was developing, majority of the codes were built around English language but as the technology progressed many nations started to get involved in this field. So, in order to avoid confusion and establish uniformity among the programming world, codes were assigned which were universal for all the languages around the globe.

We have to learn more about escape sequences and unicode if we want to avoid this kind of syntax error.

The conclusion

This article offers the most effective solution for the error: - "Unicodeescape' codec can't decode bytes in position 2-3: truncated \uxxxxxxxx escape". We discussed about the logic of interpreter and the different ways to solve this error. We strolled through the concepts of unicode and escape sequences. We also discussed the idea and purpose behind the creation of unicode.

How This Tool Improves Developer Productivity with JSON and API Workflows

Mon, 15 Dec 2025 10:12:51 +0000

JSON is the de facto data interchange format for modern web and mobile applications. However, generating, validating, and maintaining JSON structures manually can introduce friction in development workflows, particularly during rapid prototyping and API development.

Automated JSON Structure Generation

The tool converts natural-language descriptions into syntactically valid, well-structured JSON. It automatically handles:

Proper nesting of objects and arrays
List item Consistent key naming
Appropriate primitive and composite data types

This reduces human error and eliminates repetitive boilerplate.

API-Ready Response Generation

Generated output can be used directly as mock API responses. This enables frontend teams to:

Develop UI components independently of backend availability Simulate
API contracts early in the development cycle Validate response
schemas before backend implementation

This approach aligns well with contract-first and API-driven development methodologies.

Summary

By automating JSON generation and validation while supporting downstream API workflows, this tool reduces development overhead and accelerates iteration cycles, allowing engineers to focus on system design and business logic rather than data scaffolding.

Reference tool:
https://formatjsononline.com/ai-json-generator

JsonX: Lightweight JSON to C struct mapping for Embedded Systems

Fri, 17 Oct 2025 20:51:57 +0000

JsonX: Lightweight JSON → C struct mapping for Embedded Systems

JsonX is a lightweight C library for embedded developers who need to map JSON directly into structs — without dynamic memory or code generators.

JSON is everywhere - from web APIs to IoT devices.

But on microcontrollers, most JSON libraries feel like elephants in a china shop: they assume megabytes of RAM, malloc on every step, and a friendly OS.

Typical JSON libraries for embedded systems fall into two extremes:

Full-featured giants (e.g. ArduinoJson, jansson): powerful but memory-hungry.
Bare parsers (e.g. JSMN, cJSON, jfes): no mapping, just tokenization.

The Problem with JSON on Embedded Devices

When you work with RTOS or bare-metal firmware where every byte and cycle counts.
That’s where JsonX comes in - a lightweight layer over cJSON designed specifically for MCU + RTOS environments, where predictable memory and simplicity matter most.

So you either waste RAM or write tons of manual traversal code:

// classical pain
value = cJSON_GetObjectItem(root, "config");
param = cJSON_GetObjectItem(value, "baudrate");
baudrate = param->valueint;

JsonX in One Sentence

A thin mapping layer that turns JSON into C structs using a declarative table, not hand-written traversal.

Example structure:

typedef struct
{
    char      device_name[32];
    double    baudrate;
    bool      debug;
} config_t;

config_t config;

Mapping:

JX_ELEMENT config_desc[] =
{
    JX_PROPERTY_STRING("device_name", config.device_name),
    JX_PROPERTY_NUMBER("baudrate", config.baudrate),
    JX_PROPERTY_BOOLEAN("debug", config.debug),
};

const size_t config_desc_size = sizeof(config_desc)/sizeof(config_desc[0]);

And that’s it - one call:

const char *config_json_str = "{\"device_name\":\"sensor1\",\"baudrate\":9600,\"debug\":true}";

if(jx_json_to_struct(char *buffer, config_desc, config_desc_size, JX_MODE_STRICT) == JX_SUCCESS)
{
    jx_log("Configuration loaded: %s @ %lu\n", config.device_name, (uint32_t)config.baudrate);
}

No malloc() surprises, no deep tree traversal.

Core Benefits

✅ Predictable Memory
- All allocations go through RTOS pools or a static buffer.
- No hidden malloc/free. Works under ThreadX, FreeRTOS, or bare-metal.
✅ Automatic Struct Mapping
- Define once → parse many.
- Shared parsing engine for all schemas.
✅ Optional / Ignored Fields
- Unknown keys ignored automatically.
- Optional fields can have defaults.
- Makes firmware tolerant to JSON changes.
✅ Readable and Debuggable
- You still get readable JSON in logs - easier to verify on target or
  via serial output.

⚖️ JsonX vs Binary Formats

When NOT to use JsonX:

You control both ends and need max bandwidth (LoRa, NB-IoT)
You rely on schema versioning (protobuf/flatbuffers)

When JsonX shines:

You want human-readable logs
You interface with web APIs
You want deterministic memory use
You prefer C-only solution

Example: Parsing an Array of Coordinates

    #define JX_USER_BUFFER_SIZE    1024    
    // Initialize JsonX (using standard malloc/free)
    jx_init();
    
    // Define point structure
    typedef struct {
        double x;
        double y;
    } Point_t;
        
    // Define square structure with array of 4 points
    typedef struct {
        Point_t points[4];
    } Square_t;
        
        
    // Create square data structure
    Square_t square = {
        .points = {
            {0, 0},
            {1, 0},
            {1, 1},
            {0, 1}
        }
    };
        
    // Define JSON mapping for a single point object
    JX_ELEMENT point_0[] = {
        JX_PROPERTY_NUMBER("x", square.points[0].x),
        JX_PROPERTY_NUMBER("y", square.points[0].y)
    };
    
    JX_ELEMENT point_1[] = {
        JX_PROPERTY_NUMBER("x", square.points[1].x),
        JX_PROPERTY_NUMBER("y", square.points[1].y)
    };
    
    JX_ELEMENT point_2[] = {
        JX_PROPERTY_NUMBER("x", square.points[2].x),
        JX_PROPERTY_NUMBER("y", square.points[2].y)
    };
    
    JX_ELEMENT point_3[] = {
        JX_PROPERTY_NUMBER("x", square.points[3].x),
        JX_PROPERTY_NUMBER("y", square.points[3].y)
    };
    
    // Define array of point objects
    JX_ELEMENT square_array[] = {
        JX_OBJECT_VAL(point_0),
        JX_OBJECT_VAL(point_1),
        JX_OBJECT_VAL(point_2),
        JX_OBJECT_VAL(point_3)
    };
    
    // Define root object with "square" property
    JX_ELEMENT root[] = {
        JX_PROPERTY_ARRAY("square", square_array)
    };
    
    // ===== SERIALIZATION: C struct -> JSON =====
    char *json_output = jx_alloc_memory(JX_USER_BUFFER_SIZE);
    if (json_output != NULL) {
        jx_struct_to_json(root, 1, json_output, JX_USER_BUFFER_SIZE, JX_FORMATTED);
        jx_log("Serialized JSON:\n%s\n\n", json_output);
        jx_free_memory(json_output);
    }
        
    // ===== DESERIALIZATION: JSON -> C struct =====
    const char *json_input =
    "{"
        "  \"square\": ["
            "    {\"x\": 5, \"y\": 6},"
            "    {\"x\": 7, \"y\": 8},"
            "    {\"x\": 9, \"y\": 10},"
            "    {\"x\": 11, \"y\": 12}"
        "  ]"
    "}";
    
    
    // Parse JSON
    jx_json_to_struct((char*)json_input, root, 1, JX_MODE_STRICT);
    
    // Print parsed data
    jx_log("Parsed square points:\n");
    for (int i = 0; i < 4; i++) {
        jx_log("  Point %d: x=%d, y=%d\n",
                    i,
                    (int)square.points[i].x,
                    (int)square.points[i].y);
        }
    // Cleanup
    jx_parser_deinit();

Tested Configuration: STM32H7 + ThreadX (-Os, GCC 12)

⚙️ Memory & Code Footprint

Verified on STM32H7 (GCC 12, -Os, ThreadX)

jx_parser.o — 2228 B code, 4 B static RAM
Core parser and mapping logic.

jx_static_allocator.o — 248 B code, 4 B static RAM
Implements static allocator for bare-metal mode.

jx_version.o — 68 B code, 0 B RAM
Simple version string helper.

example.o (demo) — 765 B code, 1076 B RAM
Demo test including a 1 KB byte pool for ThreadX.

➡ Core only: ≈ 2.5 KB code + 8 B static RAM

➡ With example: ≈ 3.2 KB code + 1.06 KB RAM

The demo includes a 1 KB jx_byte_pool_buffer for ThreadX, but the actual pool size is user-defined.
JsonX itself has no hidden heap allocations - all memory management is deterministic.

The static allocator consumes only ≈ 12–16 bytes of header overhead inside the provided buffer.

Open Source and Final Thoughts

GitHub: https://github.com/embedmind/JsonX

JsonX is lightweight, open-source (MIT), and built purely in C for portability across RTOSes and MCU families.

Final Thoughts

This library isn’t about inventing a new protocol - it’s about making life easier for embedded developers who already have to deal with JSON.

Working with JSON on MCUs? I'd love to hear your use case - especially if you've battled with JSMN, Jansson, or ArduinoJson before.

Join the Discussion

⭐ Star on GitHub if this solves your problem
Found a bug? Open an issue

The Complete JSON & Data Toolkit

Tue, 30 Sep 2025 06:59:51 +0000

FormatJ SON Online: A Powerful JSON Toolkit That now Supports Files Up to 100 MB

**
Dealing with large JSON files can be a headache — many online tools crash or refuse files past a small size limit. That’s why I built FormatJSONOnline.com, a toolbox that not only formats, validates, and converts JSON, but now supports files up to 100 MB. Here’s what makes it stand out, and how you can get full value from it.

Key Features & Highlights
✅ Format & Beautify JSON

Clean up minified or messy JSON into a readable structure.

Live Validation

Catch syntax errors instantly — missing commas, mismatched braces, and more.

Minify JSON

Remove unnecessary whitespace to compress JSON for performance.

Conversion Tools

JSON → Excel / JSON → CSV / JSON → XML

Excel → JSON & CSV → JSON

AI JSON Generator + Mock API

Generate JSON from plain English prompts, edit it, and publish it as a live mock REST endpoint.

Support Up to 100 MB

Unlike many tools with strict size limits, FormatJSONOnline handles files up to 100 MB, making it robust for real-world API responses and large datasets.

Additional Utilities

SQL Formatter

JSON Compare / Schema Validator / Path Finder / Dummy JSON Generator

How to Use It (Step-by-Step)

Upload or Paste JSON / Prompt Description – Either bring in your own JSON or describe what you want in plain English.

Edit & Validate – Fix any errors, beautify, or minify as needed.

Convert or Publish – Convert JSON to other formats or publish it as a mock API endpoint.

Why Support for 100 MB Matters

Many online JSON tools cap at a few MB, which means you have to split, reduce, or only format parts of large responses. Supporting 100 MB means:

You can format full API responses in one go.

No need to chunk your JSON manually.

Better for real production debugging and logging tasks.

Try It Out

Give it a spin here: https://formatjsononline.com

Use the conversion pages:

Final Thoughts

Format JSON Online aims to be the one–stop JSON toolkit for developers. With support up to 100 MB, AI JSON generation, mock API creation, and robust conversions, it eliminates the need to bounce between multiple small utilities.

JavaScript Objects: Your First Step to Data Magic

Tue, 15 Jul 2025 17:39:08 +0000

Ever wondered how websites store and organize information like user profiles, product details, or your favorite playlist? The secret lies in JavaScript objects – powerful containers that make coding feel like organizing a well-structured toolbox. If you're new to programming, don't worry! By the end of this guide, you'll understand objects so well that you'll wonder how you ever coded without them.

Follow Me for More Content

Before we dive in, let's connect! I regularly share beginner-friendly coding tips and tutorials:

Portfolio: abdelhakim-baalla.vercel.app
Instagram: @abdelhakim.baalla
Twitter: @Abdelhakim99891
LinkedIn: abdelhakimbaalla

Now, let's unlock the magic of JavaScript objects!

What Are JavaScript Objects?

Think of a JavaScript object as a digital filing cabinet. Just like a real filing cabinet has labeled folders containing different documents, a JavaScript object has labeled properties containing different values.

In the real world, you might describe a person like this:

Name: Abdelhakim Baalla
Age: 19
City: Agadir, Morocco
Profession: Full Stack Developer

In JavaScript, we can represent this same information using an object:

const person = {
  name: "Abdelhakim Baalla",
  age: 19,
  city: "Agadir, Morocco",
  profession: "Full Stack Developer"
};

Objects are everywhere in JavaScript! They're the building blocks that help us organize and manage data efficiently.

Why Are Objects So Important?

Objects solve a fundamental problem in programming: organization. Imagine trying to manage information about 100 users using separate variables:

// This is messy and hard to manage!
const user1Name = "Alice";
const user1Age = 30;
const user1Email = "*Emails are not allowed*";

const user2Name = "Bob";
const user2Age = 25;
const user2Email = "*Emails are not allowed*";
// ... and so on for 100 users!

With objects, we can organize this data much more elegantly:

const users = [
  {
    name: "Alice",
    age: 30,
    email: "*Emails are not allowed*"
  },
  {
    name: "Bob",
    age: 25,
    email: "*Emails are not allowed*"
  }
  // Much cleaner and scalable!
];

Creating Your First JavaScript Object

Method 1: Object Literal Syntax (Most Common)

The easiest way to create an object is using curly braces {}:

const car = {
  brand: "Toyota",
  model: "Camry",
  year: 2023,
  color: "blue",
  isRunning: false
};

Method 2: Using the Object Constructor

const car = new Object();
car.brand = "Toyota";
car.model = "Camry";
car.year = 2023;
car.color = "blue";
car.isRunning = false;

Method 3: Using Object.create()

const car = Object.create(null);
car.brand = "Toyota";
car.model = "Camry";

Pro Tip: Method 1 (object literal) is the most popular and readable approach among developers!

Understanding Object Properties

Properties are the building blocks of objects. They consist of:

Key (property name): The label for the data
Value: The actual data stored

const smartphone = {
  brand: "Apple",        // Key: "brand", Value: "Apple"
  model: "iPhone 15",    // Key: "model", Value: "iPhone 15"
  storage: 256,          // Key: "storage", Value: 256
  hasCamera: true        // Key: "hasCamera", Value: true
};

Property Naming Rules

Use camelCase for multi-word properties: firstName, phoneNumber
Avoid spaces and special characters
If you must use spaces, wrap in quotes: "phone number"

const user = {
  firstName: "Sarah",           // ✅ Good
  lastName: "Johnson",          // ✅ Good
  "phone number": "123-456-7890", // ✅ Works but not recommended
  age: 28                       // ✅ Good
};

Accessing Object Properties

Dot Notation (Most Common)

const book = {
  title: "The Great Gatsby",
  author: "F. Scott Fitzgerald",
  pages: 180,
  genre: "Classic Literature"
};

console.log(book.title);    // "The Great Gatsby"
console.log(book.author);   // "F. Scott Fitzgerald"
console.log(book.pages);    // 180

Bracket Notation

console.log(book["title"]);    // "The Great Gatsby"
console.log(book["author"]);   // "F. Scott Fitzgerald"

// Useful when property names have spaces or special characters
const product = {
  "product name": "Laptop",
  "price-usd": 999
};

console.log(product["product name"]);  // "Laptop"
console.log(product["price-usd"]);     // 999

Dynamic Property Access

const student = {
  name: "Emma",
  grade: "A",
  subject: "Mathematics"
};

const propertyName = "grade";
console.log(student[propertyName]); // "A"

// This won't work with dot notation
// console.log(student.propertyName); // undefined

Modifying Object Properties

Adding New Properties

const pet = {
  name: "Buddy",
  type: "dog"
};

// Add new properties
pet.age = 3;
pet.breed = "Golden Retriever";
pet["favorite toy"] = "Tennis ball";

console.log(pet);
// {
//   name: "Buddy",
//   type: "dog",
//   age: 3,
//   breed: "Golden Retriever",
//   "favorite toy": "Tennis ball"
// }

Updating Existing Properties

const movie = {
  title: "Inception",
  year: 2010,
  rating: 8.8
};

// Update properties
movie.rating = 9.0;
movie.director = "Christopher Nolan";

console.log(movie.rating);   // 9.0
console.log(movie.director); // "Christopher Nolan"

Deleting Properties

const account = {
  username: "john_doe",
  password: "secret123",
  email: "*Emails are not allowed*",
  temporaryToken: "abc123"
};

// Delete the temporary token
delete account.temporaryToken;

console.log(account);
// {
//   username: "john_doe",
//   password: "secret123",
//   email: "*Emails are not allowed*"
// }

Methods in Objects

Objects can also contain functions, which are called methods:

const calculator = {
  result: 0,
  
  add: function(number) {
    this.result += number;
    return this;
  },
  
  subtract: function(number) {
    this.result -= number;
    return this;
  },
  
  multiply: function(number) {
    this.result *= number;
    return this;
  },
  
  getResult: function() {
    return this.result;
  }
};

// Using the calculator
calculator.add(10).multiply(2).subtract(5);
console.log(calculator.getResult()); // 15

Modern Method Syntax

const person = {
  name: "Alice",
  age: 30,
  
  // Modern syntax (ES6+)
  greet() {
    return `Hello, I'm ${this.name}!`;
  },
  
  celebrateBirthday() {
    this.age++;
    return `Happy birthday! I'm now ${this.age} years old.`;
  }
};

console.log(person.greet()); // "Hello, I'm Alice!"
console.log(person.celebrateBirthday()); // "Happy birthday! I'm now 31 years old."

Understanding 'this' in Objects

The keyword this refers to the object that owns the method:

const restaurant = {
  name: "Pizza Palace",
  location: "Downtown",
  isOpen: true,
  
  getInfo() {
    return `${this.name} is located ${this.location} and is ${this.isOpen ? 'open' : 'closed'}`;
  },
  
  toggleStatus() {
    this.isOpen = !this.isOpen;
    return `${this.name} is now ${this.isOpen ? 'open' : 'closed'}`;
  }
};

console.log(restaurant.getInfo()); // "Pizza Palace is located Downtown and is open"
console.log(restaurant.toggleStatus()); // "Pizza Palace is now closed"

Practical Examples

Example 1: Creating a User Profile

const userProfile = {
  firstName: "Sarah",
  lastName: "Johnson",
  email: "*Emails are not allowed*",
  age: 28,
  preferences: {
    theme: "dark",
    notifications: true,
    language: "English"
  },
  
  getFullName() {
    return `${this.firstName} ${this.lastName}`;
  },
  
  updateEmail(newEmail) {
    this.email = newEmail;
    return `Email updated to ${newEmail}`;
  },
  
  toggleNotifications() {
    this.preferences.notifications = !this.preferences.notifications;
    return `Notifications ${this.preferences.notifications ? 'enabled' : 'disabled'}`;
  }
};

console.log(userProfile.getFullName()); // "Sarah Johnson"
console.log(userProfile.updateEmail("*Emails are not allowed*")); // "Email updated to *Emails are not allowed*"
console.log(userProfile.toggleNotifications()); // "Notifications disabled"

Example 2: Building a Shopping Cart

const shoppingCart = {
  items: [],
  total: 0,
  
  addItem(name, price, quantity = 1) {
    const item = {
      name: name,
      price: price,
      quantity: quantity,
      subtotal: price * quantity
    };
    
    this.items.push(item);
    this.calculateTotal();
    return `Added ${quantity} ${name}(s) to cart`;
  },
  
  removeItem(itemName) {
    this.items = this.items.filter(item => item.name !== itemName);
    this.calculateTotal();
    return `Removed ${itemName} from cart`;
  },
  
  calculateTotal() {
    this.total = this.items.reduce((sum, item) => sum + item.subtotal, 0);
  },
  
  getCartSummary() {
    return {
      itemCount: this.items.length,
      total: this.total,
      items: this.items
    };
  }
};

// Using the shopping cart
console.log(shoppingCart.addItem("Apple", 1.50, 3)); // "Added 3 Apple(s) to cart"
console.log(shoppingCart.addItem("Banana", 0.75, 2)); // "Added 2 Banana(s) to cart"
console.log(shoppingCart.getCartSummary());
// {
//   itemCount: 2,
//   total: 6,
//   items: [
//     { name: "Apple", price: 1.50, quantity: 3, subtotal: 4.50 },
//     { name: "Banana", price: 0.75, quantity: 2, subtotal: 1.50 }
//   ]
// }

Nested Objects

Objects can contain other objects, creating powerful data structures:

const company = {
  name: "Tech Solutions Inc.",
  founded: 2020,
  address: {
    street: "123 Main St",
    city: "San Francisco",
    state: "CA",
    zipCode: "94105"
  },
  employees: [
    {
      name: "Alice Johnson",
      position: "Developer",
      salary: 75000,
      skills: ["JavaScript", "React", "Node.js"]
    },
    {
      name: "Bob Smith",
      position: "Designer",
      salary: 65000,
      skills: ["Photoshop", "Figma", "CSS"]
    }
  ],
  
  getCompanyInfo() {
    return `${this.name} was founded in ${this.founded} and is located in ${this.address.city}, ${this.address.state}`;
  },
  
  addEmployee(employee) {
    this.employees.push(employee);
    return `Added ${employee.name} as ${employee.position}`;
  }
};

console.log(company.getCompanyInfo()); // "Tech Solutions Inc. was founded in 2020 and is located in San Francisco, CA"
console.log(company.employees[0].skills[0]); // "JavaScript"

Common Object Methods

Object.keys()

Get all property names:

const fruit = {
  name: "Apple",
  color: "red",
  taste: "sweet",
  size: "medium"
};

const keys = Object.keys(fruit);
console.log(keys); // ["name", "color", "taste", "size"]

Object.values()

Get all property values:

const values = Object.values(fruit);
console.log(values); // ["Apple", "red", "sweet", "medium"]

Object.entries()

Get key-value pairs:

const entries = Object.entries(fruit);
console.log(entries);
// [["name", "Apple"], ["color", "red"], ["taste", "sweet"], ["size", "medium"]]

Practical use case - iterating through an object:

const scores = {
  Alice: 95,
  Bob: 87,
  Charlie: 92,
  Diana: 98
};

// Using Object.entries() to iterate
Object.entries(scores).forEach(([name, score]) => {
  console.log(`${name}: ${score}%`);
});
// Alice: 95%
// Bob: 87%
// Charlie: 92%
// Diana: 98%

Best Practices for Working with Objects

1. Use Descriptive Property Names

// ❌ Not clear
const user = {
  n: "John",
  a: 25,
  e: "*Emails are not allowed*"
};

// ✅ Clear and descriptive
const user = {
  name: "John",
  age: 25,
  email: "*Emails are not allowed*"
};

// ✅ Well-organized
const product = {
  id: 1,
  name: "Laptop",
  
  // Group pricing information
  pricing: {
    cost: 800,
    retail: 1200,
    discount: 0.1
  },
  
  // Group specifications
  specs: {
    brand: "Dell",
    processor: "Intel i7",
    ram: "16GB",
    storage: "512GB SSD"
  },
  
  // Group availability information
  availability: {
    inStock: true,
    quantity: 25,
    warehouse: "West Coast"
  }
};

3. Use Methods for Object Behavior

const bankAccount = {
  balance: 1000,
  accountHolder: "John Doe",
  
  // Methods for account operations
  deposit(amount) {
    if (amount > 0) {
      this.balance += amount;
      return `Deposited \({amount}. New balance: \){this.balance}`;
    }
    return "Invalid deposit amount";
  },
  
  withdraw(amount) {
    if (amount > 0 && amount <= this.balance) {
      this.balance -= amount;
      return `Withdrew \({amount}. New balance: \){this.balance}`;
    }
    return "Invalid withdrawal amount or insufficient funds";
  },
  
  getBalance() {
    return `Current balance: $${this.balance}`;
  }
};

Common Mistakes to Avoid

1. Confusing Dot vs Bracket Notation

const settings = {
  theme: "dark",
  "font-size": 16
};

// ❌ This won't work
// console.log(settings.font-size); // SyntaxError

// ✅ Use brackets for properties with hyphens
console.log(settings["font-size"]); // 16

2. Forgetting 'this' in Methods

const counter = {
  count: 0,
  
  // ❌ Missing 'this'
  increment() {
    count++; // ReferenceError: count is not defined
  },
  
  // ✅ Using 'this' correctly
  increment() {
    this.count++;
  }
};

3. Accidentally Creating Global Variables

// ❌ Missing 'const/let/var' creates global variable
person = {
  name: "John"
};

// ✅ Always declare objects properly
const person = {
  name: "John"
};

Taking Your Skills Further

Now that you understand JavaScript objects, here are some next steps to continue your learning journey:

Practice Building Real Projects: Create a todo list, address book, or simple game using objects
Learn About Classes: Objects and classes work together in modern JavaScript
Explore JSON: Learn how objects relate to JSON (JavaScript Object Notation)
Study Object-Oriented Programming: Understand inheritance, encapsulation, and other OOP concepts
Dive into Advanced Topics: Prototypes, destructuring, and spread operators

Conclusion

Congratulations! You've just unlocked one of JavaScript's most powerful features. Objects are the foundation of modern web development, and mastering them opens doors to building amazing applications.

Remember these key points:

Objects organize related data and functions together
Use dot notation for simple property access
Methods bring objects to life with behavior
The 'this' keyword refers to the object itself
Practice with real-world examples to solidify your understanding

Keep experimenting, keep coding, and most importantly, have fun with your newfound knowledge! JavaScript objects are your gateway to creating dynamic, interactive web applications.

Connect & Continue Learning

Ready to level up your JavaScript skills? I share more beginner-friendly tutorials and coding tips regularly:

Portfolio: Visit My Portfolio
LinkedIn: Connect with me
Twitter: Follow @Abdelhakim99891
Instagram: Follow @abdelhakim.baalla
GitHub: Check out my code
Dev.to: Read more articles
Medium: Follow my blog
YouTube: Watch tutorials

Found this helpful? Give it a clap, share it with fellow developers, and let me know what you'd like to learn next in the comments!

Happy coding!

Top JSON Formatting & Validation Tools (Fast + Developer-Friendly)

Mon, 14 Jul 2025 07:43:29 +0000

Here are some fast, reliable tools for JSON formatting and validation — perfect for developers working with APIs, logs, or large JSON payloads.

1. jsonformatter.org

✅ Format, validate, minify, and view JSON
Has tree view and raw text view
Supports upload + clipboard copy
Also supports XML, YAML, JWT, CSV

Fastest and most comprehensive online tool.

⚡ 2. jsoneditoronline.org

Visual + code editor side-by-side
Detects and highlights syntax issues live
Offers "Transform", "Sort", and "Filter" tools
Supports collaboration with shareable links

Great for working with deeply nested JSON visually.

3. VS Code (Built-in or Plugin)

✨ Use built-in formatting: Right click → Format Document
Extensions:

Prettier (formatting)
JSON Tools (validation, sorting, filtering)
Works offline and with huge files

Ideal if you're already coding in VS Code.

4. quicktype.io

⚙️ Convert JSON → TypeScript, Python, Java, etc.
Also formats + validates input
Instant TypeScript interfaces for Angular/NestJS devs

Best for generating types/interfaces from real API responses.

5. CLI Tools (For local work)

If you want a fast terminal-based option:

✅ `jq` (lightweight and powerful)

jq '.' data.json  # Format
jq type data.json # Validate

Install:

brew install jq   # macOS
sudo apt install jq   # Ubuntu/Debian

Fastest tool for shell scripting and JSON pipelines.

Summary Table

Tool	Format	Validate	Extras
jsonformatter.org	✅	✅	Tree view, minify, export
jsoneditoronline.org	✅	✅	Visual editing, sharing
VS Code	✅	✅	Extensions, huge JSON
quicktype.io	✅	✅	Generate TS/Java classes
`jq` (CLI)	✅	✅	Scriptable, blazing fast

How to serialize JSON API requests using Java Records?

Sun, 10 Nov 2024 05:01:58 +0000

JSON is a popularly used file format in the APIs as it is lightweight and self-describing.
JSON stands for JavaScript Object Notation and. It was derived from JavaScript and originally specified by Douglas Crockford.

JSON is easy to read, write, and language-independent, supporting data structures like arrays and objects for flexible organization. It is widely used by software teams and is a common choice for storing and transferring data. The RESTful APIs generally rely on JSON files to exchange data over HTTP.

In this tutorial blog we will learn to serialize the JSON API requests using Java Records.

What are Java Records?

A Java record is a special type of class designed specifically for working with immutable data. Java Records were introduced in Java 14 version as a preview feature and became a standard feature in Java 16.

Records offer a concise syntax for defining classes primarily intended to store immutable data.

The following is a simple example of Java Records:

public record Country(String name, long population) {
}

What is JSON Serialization?

JSON serialization is the process of transforming an object, data structure, or complex data type into a JSON (JavaScript Object Notation) format. For example, converting Java POJO into a JSON format.

During JSON serialization, each attribute of an object is converted into a JSON property, while nested structures (such as lists or other objects) are represented as JSON arrays or nested JSON objects. This helps in making the data more portable and accessible to other systems, programming languages, or APIs.

For example consider the following “Country” object using Java Records:

public record Country(String name, long population) {
}

With JSON serialization, an instance of the “Country” Record could be converted to JSON as shown below:

{
"name" : "India",
     "population":  "1.43 billion"
}

How to serialize JSON API Requests using Java Records?

API requests using request body in the JSON format can be serialized using Java Records easily through the Jackson Databind or Google Gson library. Java Records are convenient, lightweight and efficient as they automatically generate getters, as well as equals(), hashCode(), and toString() methods.

In this tutorial, we will be using the Rest-Assured library for executing the API requests and Jackson Databind library with Java Records for serializing the request body used in the API requests.

API under Test

We will be using the restful-ecommerce APIs in this blog for demonstration. It is free to use E-Commerce Web APIs for practicing API testing and is available over GitHub.

We will be using the docker-compose file to start the restful-ecommerce application and accordingly execute the API requests on our local machine.

Test Scenario

We will use the “/addOrder” API that will allow adding new orders in the system. The following is the request payload for the “/addOrder” API:

[ { 
    "user_id": "1",
    "product_id": "1",
    "product_name": "iPhone",
    "product_amount": 500.00,
    "qty": 1,
    "tax_amt": 5.99,
    "total_amt": 505.99
}]

The request accepts a JSON array with multiple JSON objects that holds the different order details.

We will serialize the JSON given in the payload using Java Records and execute the POST request to add order in the system verifying that the order is added successfully.

Configuration

We would be using the Maven project and adding the following dependencies in the pom.xml file.

<dependency>
   <groupId>org.testng</groupId>
   <artifactId>testng</artifactId>
   <version>7.10.2</version>
 </dependency>

 <dependency>
   <groupId>io.rest-assured</groupId>
   <artifactId>rest-assured</artifactId>
   <version>5.5.0</version>
   <scope>test</scope>
 </dependency>

 <dependency>
   <groupId>com.fasterxml.jackson.core</groupId>
   <artifactId>jackson-databind</artifactId>
   <version>2.18.1</version>
 </dependency>
 <dependency>

The dependencies are for TestNG that acts as a test runner, rest-assured for executing the API requests and Jackson Databind that will help in serialization.

Let’s create a new package - restful-ecommerce under the src/test folder in the project.

Let’s add two more packages under the restful-ecommerce page, the first one with name - pojo and the second one with name tests.

Test Implementation

The first step in implementation is to create Java Records with the fields needed for adding the Order as per the request payload.

The request payload requires the following fields, hence we will accordingly add those fields to the Record

user_id
product_id
product_name
product_amt
qty
tax_amt
total_amt

The data_type details for these fields can be found on the Swagger using which the fields in the Java Record can be created.

So, let’s add a new Java Record - “Order”, under the pojo package.

public record Order(String user_id, String product_id, String product_name, int product_amount, int qty, int tax_amt,
                   int total_amt) {
}

After adding the Java Record, we can straightaway move to write the API tests and create a new Java class - “AddOrderTests” under the tests package.

public class AddOrderTests {

@Test
public void testAddOrder () {

   List<Order> orders = new ArrayList<> ();
   orders.add (new Order ("1", "34", "Canon Camera CX12", 14500, 1, 915, 15415));

   given ().body (orders)
       .when ()
       .log ()
       .all ()
       .contentType (ContentType.JSON)
       .post ("http://localhost:3004/addOrder")
       .then ()
       .log ()
       .all ()
       .statusCode (201);

}
}

Code Walkthrough

The @Test annotation is from the TestNG library that will allow us to run the tests. The request payload actually accepts JSON Arrays in which multiple objects of Orders are added.

List<Order> orders = new ArrayList<> ();

The first statement in the method is creating a new list that can store Order objects. This will help us in adding multiple order objects to the list and accordingly pass them as the request body. This is done to ensure that we pass the JSON Object under JSON Arrays as the "/addOrder" endpoint needs the request body in JSON Array format.

orders.add (new Order ("1", "34", "Canon Camera CX12", 14500, 1, 915, 15415));

The second line in the method adds an object of Order with details. See how simple it is to pass the value to the Records object, Records automatically generates the required constructor for passing the values.

Next, using rest-assured methods the test for the API is written.

given ().body (orders)
   .when ()
   .log ()
   .all ()
   .contentType (ContentType.JSON)
   .post ("http://localhost:3004/addOrder")
   .then ()
   .log ()
   .all ()
   .statusCode (201);

The interpretation of the above code is pretty simple: given a body of order objects, log the request details in the console, the content-type of the request body will be in JSON format.

The Post HTTP method should be hit for the API endpoint “http://localhost:3004/addOrder”, when the API is executed, then the response details received should be logged in the console and its status code should be checked that should be equal to 201.

Test Execution

Before we execute the test, we should make sure that the restful-ecommerce application is up and running. We can start the application using the following docker compose command:

docker compose -f docker-files/docker-compose-ecommerce.yml up -d

We can then hit the following URL to make sure that the application is up and running. It should display the Swagger docs

http://localhost:3004/api-docs/

Next, we can run the test and check for the output. The following screenshots show that the JSON is serialized successfully in the API request body.

The addOrder API is successfully hit and one order is generated.

Summary

Java Records introduced in Java 16 are convenient, lightweight and efficient as they automatically generate getters, as well as equals(), hashCode(), and toString() methods.

It can be a better fit for JSON serialization and deserialization with Jackson Databind library as it does not require writing a lot of boilerplate code.

All the code examples shown in this article can be found over on GitHub.

Happy Testing!

Working With JSON File in Python

Mon, 18 Mar 2024 14:03:14 +0000

To attain the solution for data storage or integrity, handling files is significant in all programming languages, whether dealing with simple textual files or with some structured file formats like CSV and JSON or bin. These formats are used in various domains for small applications to enterprise-level programs. So, in this article, I will cover the basics of the JSON format and how Python will handle the JSON format by providing the reading and writing modules and some techniques for optimizing the performance of analyzing the JSON file.

Overview of JSON file format

Before understanding the technicality, let's first understand the format of JSON. The JSON acronym JavaScript Object Notation is a file format used for storing and transferring textual data between client and server. This format represents data in the form of key-pair value with comma separated. It also supports the nested data structures by using python objects.

{ 
  "name": "Chris", 
  "age": 23, 
  "address": { 
    "city": "New York", 
    "country": "America" 
  }, 
  "friends": { 
    "name": "Emily", 
    "hobbies": [ "biking",  "music",   "gaming" ] 
  } 
}

In this example, we have a root JSON object representing a person. It has several properties like name and age. The address property is a nested object having two properties, city and country. Then, the person object has another object representing friends having property of name and an array of hobbies including biking, music, and gaming as value. Overall, this JSON data represents the simple key-pair value structure of JSON format.

Importance and Common Use Cases

JSON is a lightweight data format used to interchange the data between client and server. It is a simple human-readable format that can be easily understood and parsed by the machine. The best use case of JSON is a web application when you are storing and transmitting data between different web-based applications, such that the client has to fill out the Contact Us form and information is sent for further processing.

The few other use cases of JSON files are:

Configuration file: Most of the developers used the JSON format to write and store the application data in the configuration file. These configuration files are easy to use because you can write all the settings and parameters of the application in them. Its major use is you don't have to change the code for a distinct environment.
Web Development: JSON data format is the best fit for storing and retrieving the data between client and server. In web development, JavaScript makes dynamic applications, whereas JSON is derived from JavaScript. So, JavaScript can easily convert the JSON object into a JavaScript object for processing.
Web API and Cross Platform: JSON format is also used for creating the web API because of its simplicity. These APIs will help to exchange the data between the web server and the client. Also, JSON is a platform-independent format and can ensure interaction with different programming languages.

Introduction to the 'JSON' modules in Python

Python provides us with a built-in module that specifically deals with JSON files. To use this module, we first have to import the JSON library and use it to manipulate the JSON file. Also, it has the specialty that we don't have to install the packages for JSON separately in the machine, but simply import the library.

Working with JSON File

While working on a program, we encountered a situation dealing with JSON data format. You must know about the ways of reading and writing operations in a JSON file. Here, the prior knowledge of file handling in Python comes in handy. So, let's start by importing the JSON module into the program.

1. Reading JSON File: using 'json.load()' function

Using the json.load() function, you read the data which is already available in the file.

import json  
file_path = "sample3.json" #provide location of file in the same directory    
with open(file_path, 'r') as file: #here, the file open operation is performed with write mode  
   content = json.load(file) #this will load the stored data from file and saved it into content variable 
   print(content)

Output

2. Parsing JSON data with error handling

Using json.loads() function, you can parse the JSON string data into a Python object. It is used when you already have the string data obtained from the API and user responses. Also, while using this function, use error handling because if the data is not in the string format, it will raise the exception to JSONDecodeError .

import json  
file_path = "sample3.json" #provide location of file in the same directory    
try: 
   with open(file_path, 'r') as file: #here, the file open operation is performed with write mode  
      json_data = '{"fruit": "Orange", "size": "Medium", "color": "Orange"}' 
      content = json.loads(json_data) 
   print(content)  
except json.JSONDecodeError as e: #it catches the exception if error occur in decoding 
   print("Decoding Error")

Output

3. Writing JSON files: Using 'json.dump()' function

Apart from reading operations, you can also perform writing operations in JSON files. By using json.dump() function, you can write all the JSON data into the file.

import json  
file_path = "sample3.json" #provide location of file in the same directory    
with open(file_path, 'w') as file: #here, the file open operation is performed with write mode  
    data = {  
       "fruit": "Apple",  
       "size": "Large",  
       "color": "Red"  
    }  
    json.dump(data, file) #it dumps the JSON data into the file using JSON dump function  
#this is used for reading operation  
with open(file_path, 'r') as file: #here, the file open operation is performed with read mode   
    print(file.read()) #print what we write in the file

Output

4. Serializing Python data structures to JSON

Using json.dumps() function, you can serialize the Python data structure i.e. list, tuples, dictionary, object, and other datatype, to JSON string that you can use for transmission and storage between client and server. It takes one argument, a Python object, and then serializes it into a JSON string. It is not used for writing to the file, but for encoding the Python objects into a JSON string.

import json  
data = {  
       "fruit": "Mango",  
       "size": "Large",  
       "color": "Yellow"  
    }  
encode_Data = json.dumps(data) #this will encode the data into string and stores in the variable  
print(encode_Data) #it prints the converted JSON string

Note: The JSON data is encoded using UTF-8 which is the default encoding in python JSON module.

Output

Handling Nested JSON Files

JSON files store the data in the string format that has the same syntax as a dictionary object in Python. However, remember that it does not work the same as a dictionary in Python. Sometimes, we get the nested JSON data during transmission and storage, where an object is contained in another object, and we have to deal with it differently. In Python, for dealing with nested JSON files, we have to navigate or iterate through the structure of dictionaries.

1. Understanding nested JSON data

To understand the nested JSON data, let's take an example where a JSON object is stored in another object having the following structure:

{ 
  "name": "Chris", 
  "age": 23, 
  "address": { 
    "city": "New York", 
    "country": "America" 
  }, 
  "friends": [ 
    { 
      "name": "Emily", 
      "hobbies": [ "biking", "music", "gaming" ] 
    }, 
    { 
      "name": "John", 
      "hobbies": [ "soccer", "gaming" ] 
    } 
  ] 
}

In the above data, you can see JSON object has a list with the name friend, where each friend is acting as a nested object with an array of hobbies.

2. Navigating and Accessing Nested Json Objects

For navigating and accessing the nested object, first, parse the Json string into the Python object, then load the data into the file. After that, you can access and navigate by using the keys and indices for each dictionary and list objects in the data.

import json  
data = '''{ 
     "name": "Chris", 
     "age": 23, 
     "address": { 
         "city": "New York", 
         "country": "America" 
         }, 
    "friends":  
    [ 
        { 
            "name": "Emily", 
            "hobbies": [ "biking", "music", "gaming" ] 
        }, 
        { 
            "name": "John", 
            "hobbies": [ "soccer", "gaming" ] 
        } 
    ] 
}''' 
encode_Data = json.loads(data) #this will parse the data and stores in the variable  
#accessing the root key 
name = encode_Data["name"] 
age = encode_Data["age"] 
#accessing the nested keys 
city = encode_Data["address"]["city"] 
country = encode_Data["address"]["country"] 
#accessing the nested data within the array 
friend1 = encode_Data["friends"][0]["name"] 
friend1_hobby = encode_Data["friends"][0]["hobbies"] 
print("Name is:",name) 
print("Age is:", age) 
print("Country:", country) 
print("City:", city) 
print("Friend name:",friend1) 
print("Friend hobby: ", friend1_hobby)

Output

3. Techniques for flattening JSON data

Flattening is the technique that is used to make the nested structure of Json data into tabular format. This format makes it easy to analyze and manipulate the data. Pythons provided us with a library named “pandas” that gave us a simple function json_normalize() for flattening the nested structure of JSON data into a data frame.

import json 
import pandas as pd #import pandas library 
data = '''[ 
  { 
    "name": "Chris", 
    "age": 23, 
    "address": { 
      "city": "New York", 
      "country": "America" 
    } 
  }, 
  { 
    "name": "Emily", 
    "age": 25, 
    "address": { 
      "city": "New York", 
      "country": "America" 
    } 
  }, 
  { 
    "name": "John", 
    "age": 27, 
    "address": { 
      "city": "New York", 
      "country": "America" 
    } 
  } 
]''' #nested data 
encodedata = json.loads(data) #first parse the json string into python object  
flaten_data = pd.json_normalize(encodedata) #then, flaten the data into tabular format 
print(flaten_data) #print the table

Caution: Remember, not all python datatypes converted into JSON i.e. tuples are converted into list.

Output

Performance Optimization and Best Practices

While dealing with Json files, several performance considerations need to be followed to ensure the robustness and maintainability of the program. Here, I am going to discuss some performance and optimizing techniques and the best practices for writing clean and efficient code.

Performance Considerations for working with large JSON

Memory constraints: Large Json files take a lot of memory while loading into the RAM. So, it's better to check the size of the file and available resources before processing it.
Streaming Technique: if you are dealing with large Json files, it is preferred to use chunks for processing the file. Large Json files will consume a lot of memory when we load the file in the RAM. So, using the Json streaming technique will load the file in the form of chunks rather than loading the entire file.
Incremental approach: Python provides us with a library with the name ijson, this library allows you to parse the large Json file iteratively. It's better to use libraries instead of creating chunks in the program.

Techniques for optimizing JSON file operation

Efficient serialization: while working with Json file parsing use the efficient libraries cjson or ujson. These libraries provide a much more efficient way of serialization and deserialization of Json data instead of standard libraries.
Usage of Api and generator function: when working on a large Json file, try to use the streaming API or some generator function that reads the Json objects line by line instead of loading them into the memory at once.
Caching: if you are accessing the same Json file frequently then increase the usage of the cache by reducing the data loading request again and again.
Processing: if you are working with many files, try to use batch processing instead of working line by line. This helps you to improve the performance of input-output operations.

FAQs Q: Is dump() and dumps() work same?
A: No, the dump() function is used to serialize the python object to json string and store it into the file, whereas the dumps() works the same but is not stored in the file Q: What is the ijson library?
A: The ijson library is used for iteratively parsing the json file. It deals with larger json files.

Wrapping Up

In conclusion, you saw that JSON is a platform-independent format used for interchanging the data between client and server. Python provides us with various powerful built-in tools and libraries to deal with Json files. Whether you are developing a web application or working on web API, understanding the fundamentals of json is significant. It is a versatile format that can work with any programming language. So, mastering the methods mentioned in the article will help you to deal with json files efficiently. I hope this guide is helpful for you. Happy Coding!

Reference

Working with Json

Json.decoder.jsondecodeerror: expecting value: line 1 column 1 (char 0)

Thu, 19 Oct 2023 11:37:31 +0000

This article concerns what "JSON decoder error" is and how to fix it. In the world of JSON data handling, you come across many errors but JSON decoder error is one of the common ones. This error often arises when the input JSON data is either improperly formatted or missing entirely, leaving developers scratching their heads. Carefully reviewing your JSON data format can help you successfully parse it, improving your data handling skills. We'll explore why this error occurs and strategies to rectify it. So, let's embark on this journey to conquer the JSON decoding challenge.

What is a JSON decoder function?

A JSON decoding function converts a JSON-formatted string into a programming language-manipulable data structure. These functions are frequently encountered in programming libraries. In order to make working with the data within your code easier, it takes a JSON string as input and converts it into a data structure, such as a dictionary (in Python), an object (in JavaScript), or a similar data structure in other languages. JSON decoding functions are used in many programming languages to convert JSON data to native data structures.
When you need to work with data from outside sources, such as APIs, or when you need to parse configuration files in JSON format, JSON decoding routines come in handy. They ensure the JSON data is appropriately parsed and can be used as native data types in your programs.

What does the 'json-decode-error' mean?

Error:
Json decode error

When trying to decode (parse) a JSON-formatted string into a data structure but the input JSON data is not well-formed or contains syntax mistakes that preclude proper parsing, the "JSONDecodeError" error occurs. The JSON data provided is not in a proper format, according to this issue, which commonly occurs when working with JSON data in a programming context.
In order to resolve this error, you need to review and fix the issues in the JSON data to ensure it conforms to the JSON syntax rules, which we will be discussing further in this blog.

A few common scenarios that can lead to this error:

There can be multiple reasons for this error to occur, but here are a few most common ones:

Scenario1: Parsing an Empty string

When you attempt to parse an empty string using a JSON decode, such as the json.load() function in Python, you encounter the JSONDecodeError: Expecting value: line 1 column 1 (char 0) error. This error arises because an empty string essentially lacks any JSON data to work with. An empty string provides no such structure or data content to interpret, making it impossible for the JSON decoder to identify and convert it into a meaningful data structure. Therefore, the error message serves as a signal that the JSON decoder expects to find a valid JSON value at the start of the string but instead encounters a lack of content.

Example:

import json

empty_string = ""
try:
    data = json.loads(empty_string)
except json.JSONDecodeError as e:
    print(f"JSONDecodeError: {e}")

Output:

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Scenario2: Incorrectly formatted JSON

JSON is a highly structured data format that adheres to predefined guidelines for information representation. The warning JSONDecodeError: Expecting value frequently denotes problems with JSON data formatting. It's vital to make sure that your input string complies with these guidelines to prevent this issue. One of the following valid JSON values should always start JSON data:
**- Object

Array
String
Number
Boolean
Null**
A JSON data structure should initiate with one of these valid JSON values to be correctly formatted.

Example

import json

invalid_json = '{"name": "John", "age": 30,}'  
try:
    data = json.loads(invalid_json)
except json.JSONDecodeError as e:
    print(f"JSONDecodeError: {e}")

Note: Incorrect format due to an extra comma in 'invalid_json' variable.

Scenario3: Network or File reading issues

Sometimes accessing data from external sources you end up getting this error. If you are trying to parse JSON data from an external source, such as an API or a file, and the source is returning an empty response or an invalid response, it can trigger this error.

Example

import json

# Assuming 'empty.json' is an empty file
with open('empty.json', 'r') as file:
    json_data = file.read()
    try:
        data = json.loads(json_data)
    except json.JSONDecodeError as e:
        print(f"JSONDecodeError: {e}")

Solution

Scenario 1: Solution: Parsing an Empty string

Before attempting to parse an empty string, you must first determine whether it is empty. An easy conditional statement can accomplish this. For instance, Python:

import json

input_string = ""  # Your input string
if input_string:
    data = json.loads(input_string)
else:
    # Handle the case when the string is empty
    print("The input string is empty.")

By this, you can easily identify the error or mistake and take action to debug it.

Scenario 2: Solution: Incorrectly formatted JSON

You must make sure that the input string is genuine JSON if you want to fix problems with improperly formatted JSON. This can entail fixing any syntax issues or missing JSON data pieces. To help find and correct formatting mistakes, you can also use online linting or JSON validation tools.

Scenario 3: Solution: Network or File reading issues

Consider the following when working with JSON data from external sources, such as files or APIs:

Verify that the source is providing data.
Make sure that the file or API is not empty.
Use error handling methods to handle potential network problems or file reading errors compassionately. Use try-except blocks, for instance, in Python to catch exceptions and deal with them properly.

Here's an example of handling network or file reading issues:

import json
import requests

try:
    response = requests.get('https://example.com/api/data')
    response.raise_for_status()  # Check for HTTP errors
    json_data = response.text

    if json_data:
        data = json.loads(json_data)
    else:
        # Handle the case when the response is empty
        print("No data received from the API.")
except requests.exceptions.RequestException as e:
    # Handle network errors
    print(f"Network error: {e}")
except json.JSONDecodeError as e:
    # Handle JSON decoding errors
    print(f"JSON decoding error: {e}")

By using these fixes, you can handle situations where the JSONDecodeError might occur efficiently and make sure your code gracefully handles empty strings, improperly structured JSON, or problems with external data sources.

Conclusion

In conclusion, it's critical to address the fundamental causes methodically when dealing with JSONDecodeError: Expecting value: line 1 column 1 (char 0) or similar problems. Take the following essential actions to fix these mistakes and guarantee flawless JSON data handling:
**- Validate Source

String Integrity
By following these guidelines, you may confidently manage the complexities of JSON decoding, efficiently handle errors, and take advantage of the capabilities of structured data interchange for your applications. With these techniques, you'll be well-equipped to tackle** typical challenges in your coding travels. JSON decoding, like coding itself, is a voyage of accuracy and problem-solving.

References

JSON Decode Error

Coder Legion - Recent posts tagged json

Three ways a jq one-liner lies to your bash script

The Trigger Swap That Broke Every Downstream Reference

Why Your LLM Still Returns Broken JSON

The Three Workarounds We All Wrote First

What Constrained Decoding Actually Does

JSON Mode Is Not Schema Enforcement

What It Still Does Not Give You

JsonToolBox

Replay a Flight After It Lands: Historical Flight Tracker API

What The API Returns

Example Request

Example Output

Fetching Historical Flight Data in JavaScript

Extracting the Flight Path

Finding the Max Altitude

Example Use Cases for Past Flight Track Data

Bringing Past Flights Back to Life - All With Data!

A JSONPath engine that follows the spec and can't run code

The standard nobody had

The filter that runs a shell

A scout that stays on the path

The parts of the spec worth carrying around

The part I tried to borrow

Why it can't be turned into code

Is it the right tool?

Getting started

Closing thoughts

Working with nested JSON: exploring it, mocking it, and querying it in Postgres

1. Reading a huge nested payload

2. Generating realistic nested JSON, fast

3. Querying and updating nested JSON in Postgres

Where the tool fits

How to View and Analyze Large JSON Files Without Getting Lost

Why Large JSON Files Are So Hard to Read

Common Challenges When Working with Large JSON Datasets

Getting Oriented: Understanding JSON Structure

The Power of JSON Tree View

Techniques for Navigating Nested JSON Structures

How to Search Within Large JSON Files

Analyzing API Responses That Span Thousands of Lines

Reviewing Config Files and Exported Datasets

Common Mistakes to Avoid

Practical Tips That Save Real Time

Conclusion

Why Your React Frontend Crashes When an LLM Streams Malformed JSON

react #nextjs #typescript #ai #frontend #webdevelopment #javascript #softwareengineering #llm #sse

fastjson 0.3.0: A Faster Drop-In ext/json for PHP, Backed by yyjson

Why drop in faster JSON for PHP

What "drop-in" actually means here

The numbers

How the encoder gets to 6×

The memory tradeoff

Compat harness

Upstream collaboration with yyjson

Install

The four-character migration

Parsing hundreds of megabytes of JSON in milliseconds in React Native

What JSON.parse actually does in React Native

Benchmark fixture (source)

Measured results (parseFile, release builds, physical devices)

Why “read file as string, then JSON.parse” can be worse at peak

The technique: how react-native-fast-json does it

1. Native buffer (simdjson)

2. Lazy JsonView (Nitro hybrid object)

3. Materialization is explicit and expensive

Architecture

What you still pay for (honest accounting)

Why milliseconds (or low hundreds of ms) are plausible for hundreds of MB

Comparison table (same fixture, measured + observed)

Wildcards and paths

When native lazy parsing is the wrong tool

Not a common use case, prefer better alternatives

When this library still does not help enough

Benchmark methodology

Summary

Try react-native-fast-json

Don’t Just Ask AI for JSON. Design the Output.

The three ways structured output often breaks

1. Text before the JSON

What `JSON.parse` actually does in React Native

Measured results (`parseFile`, release builds, physical devices)

Why “read file as string, then `JSON.parse`” can be worse at peak

2. Lazy `JsonView` (Nitro hybrid object)