Coder Legion - Recent posts tagged mlops

Post-Mortem Analyses of Publicly Known AI Security Incidents: Key Architectural Insights

Mon, 12 Oct 2026 10:09:49 +0000

As artificial intelligence systems transition from isolated proof-of-concept models to deeply integrated production architectures, their attack surface has undergone a fundamental shift. Conventional software security paradigms—which focus primarily on memory safety, injection vulnerabilities, and role-based access control—are inadequate for systems based on probabilistic outputs, unstructured inputs, and vast parameter spaces.

The investigation of publicly disclosed AI security incidents provides crucial empirical data for hardening modern machine learning pipelines. The following section analyzes recurring classes of vulnerabilities, their systemic causes, and architectural safeguards from a technical perspective.

Taxonomy of Vulnerabilities in AI Production Systems

A. Prompt Injection & Indirect Control Flow Hijacking

Mechanism: Untrusted external data (e.g., crawled web pages, emails, or user-uploaded files) contain instructions designed to bypass system prompts or the control mechanisms of autonomous agents.
Incident profile: LLM-based agents execute external API calls or apply Retrieval-Augmented Generation (RAG) to malicious datasets. When the model processes untrusted tokens, it interprets them as executable instructions rather than mere data.
Cause: Failure to maintain a strict separation between the instruction context (system/developer prompts) and the data context (user inputs and retrieved documents).

B. Training Data Poisoning & Supply Chain Compromise

Mechanism: Adversarial manipulation of datasets or pre-trained base models prior to fine-tuning or deployment.
Incident profile: Attackers inject subtle backdoors or trigger patterns into open-source datasets. This causes the models to behave securely on standard evaluation benchmarks while failing catastrophically under specific trigger conditions.
Cause: Inadequate provenance tracking, lack of cryptographic verification of training artifacts, and absence of automated anomaly detection for training corpora.

C. Modeling of Infiltration, Extraction, and Inversion

Mechanism: Systematic querying of proprietary models to reconstruct proprietary weights, extract training data, or infer hyperparameters.
Incident profile: Compromise of commercial APIs via mass side-channel queries that map decision boundaries or expose personally identifiable information (PII) embedded during training.
Cause: Excessively permissive rate limiting, absence of output perturbation or differential privacy mechanisms, and unlimited granularity of API responses.

Structural Root Causes

The analysis of these incidents reveals three systemic patterns of failure in technical organizations:

The "software vs. statistics" category error: treating ML models as deterministic components rather than as non-deterministic statistical systems. Since models execute semantic logic rather than syntactic rules, conventional input validation (such as sanitization using regular expressions) falls short when dealing with semantic payloads.
Unrestricted operational authority for agents: Granting LLM agents extensive, unrestricted capabilities to use tools (e.g., shell access, unvalidated database write permissions, or unrestricted HTTP clients) without intermediary human validation or limiting access rights to the necessary minimum (principle of least privilege).
Blind spots in the supply chain: reliance on unverified model checkpoints and third-party datasets downloaded from public platforms—without robust artifact scanning and behavior-based regression testing.

Engineering Playbook: Architectural Safeguards

To develop a resilient AI infrastructure, teams must implement a multi-layered defense strategy (defense-in-depth) across the entire lifecycle:

Dual-model protection mechanisms & input sanitization: Implement intermediate classification layers (smaller, highly specialized security models) to evaluate incoming data before it reaches the primary processing mechanisms. Treat all data chunks retrieved via RAG as untrusted input.
Isolation of structural context: Leverage new API primitives and token-level structural markers that enforce strict separation between system logic and dynamic user data (e.g., explicit delimiters for chat templates and role isolation).
Execution of tools based on the principle of least privilege: Agentic tool environments run in a sandbox. API calls are embedded in deterministic middleware that validates parameters against strict JSON schemas and business logic constraints prior to execution.
Artifact Provenance & SBOMs for AI: Implement Software Bill of Materials (SBOM) standards for datasets and model weights. Use cryptographic hashing (SHA-256) and secure artifact registries to track every stage of the MLOps pipeline.
Continuous Red Teaming & Behavioral Fuzzing: Integrate automated adversarial testing (e.g., automated prompt injection fuzzers) into CI/CD deployment pipelines to detect regressions in the model's security boundaries prior to production deployment.

Conclusion

Securing AI systems is an ongoing process; it requires a synthesis of traditional cybersecurity disciplines and novel, ML-specific threat models. As AI engineers and technical architects, our task is to develop systems that anticipate adversarial manipulation by default and shift our approach from reactive patching to proactive, demonstrable resilience.

What architectural patterns has your team implemented to minimize risks from prompt injection and supply chain attacks in production? Let’s discuss this in the comments.

LLM Serving on Kubernetes in 2026: What's Solved and What's Still Open

Sat, 10 Oct 2026 16:09:22 +0000

A field survey of the Kubernetes + LLM inference stack in 2026 — the problems that now have serious players, and the edges that are still genuinely underserved.

If you run large language models on Kubernetes, you've probably noticed the ground shifting fast. A year ago most of the hard problems were open. In 2026 a lot of them have serious, well-funded players — which changes the calculus if you're deciding where to build. Competing with a GA project is very different from pioneering an empty space.

This is a map of the territory: what's crowded, and where I think the real gaps still are. I'm deliberately not pitching any of these as "my novel architecture" — several have academic papers or shipping projects attached. The point is to show where the open problems are so you can decide where your effort pays off.

What's already crowded

These are the areas where the obvious pain points now have mature answers. Building here means competing, not pioneering.

KV-cache-aware routing / model-aware gateways. Largely solved. The Gateway API Inference Extension (a kubernetes-sigs project) defines a standard InferencePool routing contract, and llm-d, NVIDIA Dynamo, and GKE Inference Gateway (which went GA in 2026) all do prefix-cache affinity and cache-aware routing on top of it.

Prefill/decode disaggregation. NVIDIA Dynamo and llm-d cover this well.

Fractional GPU sharing (MIG / MPS / time-slicing). Run:ai, cast.ai, the NVIDIA GPU Operator, and vCluster all address it.

GPU scheduling primitives (DRA, gang scheduling). KAI Scheduler, Grove, Kueue, and Volcano are all in play, with Dynamic Resource Allocation graduating in core Kubernetes. Crowded, though still messy at the edges.

If your idea lives in one of those buckets, go in with eyes open: you're entering a race.

Where the real gaps still are

Three edges still look genuinely underserved, and each is shaped like a Kubernetes-native problem — a CRD plus a controller plus some node-level machinery.

1. Model-weight distribution as a first-class Kubernetes primitive

This is the one I find most interesting. Model weights are often 100+ GB, and today they're still delivered largely by ad-hoc downloads from object storage — without the pull-caching, digest addressing, and content verification that Kubernetes already gives you for container images. That gap is a major driver of the two-to-ten-minute cold starts that make scale-to-zero economically painful for inference, since every new replica pays that cost before serving its first request.

The literature backs up the problem. An empirical study (arXiv 2607.16596, "Cold-Start Model Delivery in Kubernetes Inference Serving") measures the delivery paths — modelcar sidecars, native image volumes, object-storage download — on artifacts sized to 1B-, 7B-, and 70B-class weights, and a companion paper (arXiv 2609.20874) frames the same two-to-ten-minute startup as the reason reactive autoscaling is structurally late. So the pain is validated; what's missing is a first-class primitive for it. Note that these papers study and characterize the problem — they don't ship the CRD-shaped solution below, and neither do I claim to have invented it.

The interesting shape of a solution: a ModelArtifact / ModelCache CRD, a controller, and a node-level DaemonSet that treats weights like OCI-addressable, content-verified, node-locally-cached blobs — with topology-aware peer-to-peer prewarming (pull from a neighbor node's cache over the fast fabric instead of re-pulling from object storage), plus scheduler hints so pods land where the weights already live. NVIDIA Dynamo's ModelExpress already does some of this (one worker publishes weights, others pull over NIXL/RDMA), so even here you'd be refining a direction others have started, not working in a vacuum.

Why it's a good build target: it's infrastructure-shaped, CRD-heavy, and not yet a solved commodity. You can prototype the caching, P2P, and scheduler-hint logic and measure cold-start reduction on modest hardware — or even with simulated weights — without a large GPU budget.

2. SLO-driven inference autoscaling that understands request heterogeneity

A caution up front: "SLO-aware inference autoscaling" as a headline is already a shipped commodity — AWS, KEDA, and academia all have versions of it. So this is only interesting if you go narrower than the headline.

Current autoscaling scales on queue depth or KV-cache utilization. But the Cascade paper (arXiv 2608.06557, "Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving") shows that requests under the same SLO can differ in urgency by a wide margin — a short chat versus a long agentic reasoning trace — and that you can direct queueing and data-movement overhead toward the requests that can absorb it. Related systems like Scorpio (arXiv 2505.23022) exploit the same SLO heterogeneity for admission and scheduling, so this is an active research line, not an empty one. The defensible wedge is per-request-class latency budgeting plus admission control: an InferenceSLO CRD where you declare per-class latency budgets, and a controller that combines a batch-latency predictor with HPA/KEDA to make admission and scaling decisions on predicted goodput rather than raw queue length.

This one is more research-flavored and genuinely publishable — but only if you build and benchmark it against the existing SLO-aware baselines. Without numbers, it reads as a restatement of what's already shipping.

3. Multi-LoRA adapter lifecycle as a CRD

Serving hundreds of tenant-specific LoRA adapters over one base model is now a real production pattern (arXiv 2511.22880), and vLLM and Dynamo support runtime adapter loading. What's still hand-rolled is lifecycle management: which adapters are hot, cache eviction by usage, routing requests to workers that already hold the adapter, and VRAM budgeting. Dynamo has some declarative Kubernetes support here, but it's early.

The shape: a LoRAAdapter + AdapterPool CRD that manages adapter placement, usage-based eviction, and adapter-affinity routing hints — essentially "KV-cache-aware routing," but for adapters.

How I'd choose

If you want a portfolio or OSS operator project: idea #1. It's clearly the most infrastructure-shaped, there's a fresh paper validating the problem, and it's demonstrable without a big GPU budget.

If you'd rather write something paper-worthy: idea #2, with the per-request-class framing and real benchmarks.

Idea #3 is a reasonable middle ground.

A closing note on honesty, because it matters in this space: none of these are empty green fields. Each sits next to active prior art, and readers who work on inference will know it. The value here isn't a claim of novelty — it's a clear-eyed map of where the race is already crowded and where there's still room to build something useful.

Sources

Gateway API Inference Extension — kubernetes-sigs/gateway-api-inference-extension (github.com)
NVIDIA Dynamo; ModelExpress weight distribution (docs.nvidia.com)
Run:ai, cast.ai, NVIDIA GPU Operator, vCluster (GPU sharing)
KAI Scheduler, Grove, Kueue, Volcano; Dynamic Resource Allocation in Kubernetes
arXiv 2607.16596 — "Cold-Start Model Delivery in Kubernetes Inference Serving: An Empirical Study of OCI-Based Distribution and Its Integrity" (G. Kliukovkin)
arXiv 2609.20874 — "Decomposing Predictive Kubernetes Autoscaling for LLM Serving Under Long Startup Delays"
arXiv 2608.06557 — "Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving"
arXiv 2505.23022 — "Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference"
arXiv 2511.22880 — "Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems"

Agent Evals in Practice: Offline, Online, and Braintrust vs LangSmith vs Datadog

Mon, 28 Sep 2026 05:43:07 +0000

Originally published on thomaskalnik.com.

I have set up evals for two very different systems. The first was the chat agent in Spotter Studio, a now-retired tool that helped YouTube creators come up with video ideas and titles. The evals there covered routing and behavior: did a request go to the right part of the agent, and did the agent then do what it was supposed to do. The second was a retrieval system at a nonprofit, where the question was whether semantic search over social posts and media returned the right items, and whether a new embedding model was actually better than the old one.

Both used Braintrust. I have also used LangSmith and Datadog's LLM Observability. This post covers what I would set up again, the mistakes worth skipping, and how the three platforms compare today.

What an eval is made of

Every eval platform uses the same four parts, with different names:

Dataset: inputs, and where possible the expected output or the facts a good output must contain.
Task: the code under test. It takes one input and returns an output: a route, a tool call, an answer, a ranked list.
Scorers: functions that turn an output into a number. Some are plain code (exact match, recall@k, "did it call the tool with a valid argument"). Some are an LLM judge with a rubric. Some are a human.
Experiment: one run of the task over the dataset with a fixed version of the code, prompt and model, stored so it can be compared with the next run.

The dataset and the scorers are the part that takes real work. The platform's job is to make experiments cheap to run and easy to compare.

Offline evals for an agent

Offline evals run before a change ships, against a fixed dataset. For a chat agent I split them into three layers, cheapest first.

Routing evals

A creator-facing chat agent gets many kinds of requests: brainstorm video ideas for a channel, rewrite a title, explain why one title might work better than another, or something the agent should not attempt at all. A router decides which flow, tool or sub-agent handles each one. If routing is wrong, nothing downstream can fix it, so this is the first thing to test.

Routing evals are the easiest evals you will ever write. The dataset is a list of user messages, each labeled with the route it should take. The scorer is exact match. A case looks something like this (illustrative, not a real Spotter case):

{
  "input": "my last video on budget travel flopped, give me 5 better title options",
  "expected": { "route": "title_generation" },
  "metadata": { "source": "synthetic", "tricky": "about a past video, but asks for titles" }
}

Three things make these evals useful:

Collect cases from production, not from imagination. The messages that break a router are the ambiguous ones: a request for ideas that also asks for titles, or a follow-up that only makes sense given the previous turn. You find those in real conversations.
Look at the confusion matrix, not only the accuracy. One number going from 91% to 93% tells you little. Knowing that "title rewrite" keeps getting sent to "idea generation" tells you what to fix.
Run them on every change. The scoring is deterministic, but the router is a model call, so it still costs something and can vary between runs. Include the previous turns for follow-up cases, and repeat borderline cases a few times before making them a blocking check on pull requests.

Behavior evals

Once the request reaches the right place, the question becomes whether the agent behaved well: did the ideas fit the channel, were the titles in the requested style and length, did it ask a clarifying question when it should have, did it avoid promising things the product could not do.

Most of this needs an LLM judge. The practices that make a judge useful:

One rubric question per scorer. "Is every title under 70 characters" is plain code. "Does each idea fit the channel description" is a judge. A single judge asked to rate "overall quality" produces a number nobody can act on.
Pin and version the judge. The judge's prompt and model are part of the experiment. If they change, the scores are no longer comparable with last week's.
Calibrate the judge against people. Label a few dozen outputs yourself, run the judge on the same outputs, and read the disagreements. You will usually find the rubric is ambiguous, not that the judge is bad.
Prefer pairwise comparisons for subjective output. "Is version B's set of titles better than version A's for this channel" is easier to judge reliably than an absolute score out of 10.

Trajectory checks

For an agent that calls tools, the final answer is not the only thing that matters. Useful code scorers check the path: the tool that was called, whether its arguments were valid, how many steps it took, and whether it did anything it must never do. These are cheap and catch a surprising number of regressions after a prompt change.

What a retrieval eval taught me about eval hygiene

The nonprofit's retrieval eval was not an agent eval, but it taught me the most about keeping an eval honest. The existing setup was a test in the repo's test runner: 20 questions with hand-labeled correct answers, a corpus of a few thousand text posts and media items, and a GitHub Action that posted precision, recall and F1 to the pull request when a label was added.

It had a quiet bug in its design. The runner retrieved exactly as many items as each question expected, so precision, recall and F1 were always the same number. That is mathematically expected when k equals the number of relevant items, so nothing was broken in the code. But it meant the three metrics could not move independently, and the eval said nothing about what happens with a fixed retrieval budget. Before trusting any eval, check that each metric can actually disagree with the others.

What I changed, and would do again:

Two jobs, two tools. I kept the test in the test runner as the hard gate: below a set score, the pull request fails. Braintrust ran alongside it for everything the gate is bad at: side-by-side comparison of embedding models, per-question drill-down, and history across pull requests. A CI gate should be simple and boring. A place to explore results should be rich. Keeping them separate meant neither had to compromise.
Put the experiment's identity in its name and metadata. Each run was named from the provider, model and embedding dimension, with metadata for the corpus type and k. Comparing "model A at 1024 dimensions, text only" with "model B, images included" is then a filter, not an archaeology project.
Cache the expensive part. Embedding the whole corpus is slow and costs money. Caching the embeddings meant reruns only redid retrieval and scoring, which made iterating on the questions and scorers take seconds instead of minutes.
Drill into single cases. The per-question view showed that text posts were retrieved reasonably well while media items were found much less often. One likely cause was easy to spot once visible: media had been embedded from its text metadata only, not from the images. An average score would never have shown that.

The scorers went through a lesson of their own. I started with about fourteen: precision, recall and F1 for each item type, NDCG, MRR, and similarity diagnostics such as the average similarity of correct and incorrect hits and the gap between them. One of them could go negative, and Braintrust expects scores between 0 and 1, so I clamped it: a workaround that throws information away, so keep the raw value somewhere as a diagnostic. Within a day I cut the list to eight: recall and mean average precision for each item type, hit rate for media, the average similarity of correct hits for each item type, and one combined score. Fourteen numbers per run meant nobody looked at any of them. Eight, each tied to a decision, got read.

Online evals

Offline evals only cover the cases you thought to collect. Online evals score real production traffic, after the fact, and they are how the dataset keeps growing.

The pattern that works:

Run cheap code checks on all traffic and LLM judges on a sample. Formatting, length, refusals, tool errors and latency can be checked on every trace. How much to send to a judge depends on volume, how rare the failures you care about are, and which segments matter most. Sample more heavily where failures are rare or costly.
Score asynchronously. Online scoring must never add latency to the user's request.
Alert on the trend, not on single traces, except for severe failures such as leaked data or unsafe output, which deserve an alert every time. For everything else, a drop in the daily share of conversations the judge rates as on-topic is a signal; one bad trace is not.
Close the loop. The most valuable output of online evals is new offline test cases. When a production trace fails, add it to the dataset with the correct label, and the next change is tested against it.
Mind the data. Production traces contain user data. Check what the platform stores, for how long, and where, before turning on full capture, and redact before copying production traces into a dataset.

Braintrust vs LangSmith vs Datadog

I have used all three, and Braintrust is my favorite. Apart from that preference and the Braintrust details from my own projects, what follows comes from a fresh read of each vendor's documentation and pricing page in September 2026. These products ship quickly, so check anything that matters to you against the current docs.

Braintrust

Braintrust is the one I would pick again. Its core is the experiment loop: you write an Eval() with data, a task and scorers, run it, and every run becomes an experiment you can compare against a baseline, with a per-example diff and each case marked as an improvement, a regression or a tie. The per-question view in that comparison is where the media-retrieval gap above became visible.

Other things that matter in practice:

Scores are numbers between 0 and 1. That is a small constraint (I had to clamp a scorer that could go negative), but it keeps every scorer on the same chart scale.
Autoevals, an open-source library of ready-made scorers (factuality, similarity, JSON validity and others), plus LLM-judge and plain-code scorers.
An official GitHub Action that runs evals on a pull request and posts the results as a comment.
Online scoring through rules that run scorers asynchronously on a sample of production spans or whole traces.
Agent guidance in the docs that covers routing ("did it go to the correct model or sub-agent"), tool choice and tool arguments. It has no packaged trajectory matcher or simulated-user feature. You write those as scorers.
No per-seat pricing, which matters when you want product managers and designers in the tool reading results.

The weaknesses: the platform is closed source, self-hosting is Enterprise-only, and scores are metered. If you run judges on a lot of production traffic, the score count is the number to watch.

LangSmith

On paper, LangSmith has the most built-in tooling for agent-specific evaluation, and its own libraries do more of the work for you. agentevals gives you trajectory matching in four modes (strict, unordered, subset, superset), openevals can simulate a user for multi-turn tests, and the docs have a dedicated guide for single-step routing evals. Online evaluators can have a weekly spend cap, which is a thoughtful feature for LLM judges on live traffic, and alerts can go to Slack or PagerDuty.

It works without LangChain, through its own SDK or OpenTelemetry, and it integrates most deeply with LangChain and LangGraph apps. For CI it offers pytest and Vitest/Jest plugins rather than a packaged GitHub Action. Pricing is per seat plus per trace, and the overage rate sits behind a calculator rather than on the page. Having used both, I still prefer Braintrust.

Datadog LLM Observability

Datadog's product, which it is now also calling Agent Observability, has grown a full eval toolset: versioned datasets, experiments with side-by-side results, LLM judges at span, trace and session level, nine judge templates including tool selection and tool-argument correctness, annotation queues (since March 2026), and a prompt registry. Since June 2026 it can also run DeepEval and Pydantic Evals metrics unchanged.

Its real advantage is everything around the evals. LLM traces sit next to your APM traces, logs and real-user monitoring, and eval results can drive the same monitors your on-call rotation already uses. Its pricing is also the simplest of the three: you pay per LLM call span, and tool and retrieval spans are free. There is no separate evaluation fee, but the LLM calls a judge makes count as billable spans.

Where it is weaker: it started as an observability product, and most of its offline tooling (datasets, experiments, annotation queues) is newer than the other two platforms'. I found no packaged eval action for CI in the docs, so pass/fail thresholds in a pipeline need your own integration code around the SDK. It is SaaS only.

Feature comparison

	Braintrust	LangSmith	Datadog
Framework-agnostic SDKs	Python, TS, Go, Java, Ruby, C#	Python, TS	Python, Node, Java
OpenTelemetry ingest	Yes	Yes	Yes
Datasets from production traces	Yes	Yes	Yes
Experiment comparison	Per-example diff, regressions flagged	Diff mode for two runs	Side by side
Code and LLM-judge scorers	Yes, scores in [0, 1]	Yes, any range	Yes, typed outputs
Trajectory / tool-call evals	Custom scorers	`agentevals`, four match modes	Tool selection and argument templates
Simulated users	Not built in	Yes (`openevals`)	Not built in
Online evals with sampling	Yes	Yes, with spend caps	Yes
CI	Official GitHub Action	pytest / Vitest plugins	Your own code around the SDK
Human review queues	Yes	Yes, including pairwise	Yes
Self-hosting	Enterprise only	Enterprise add-on	No

Pricing, as of September 2026

	Free tier	Named paid plan	What you pay for
Braintrust	1 GB of data, 10k scores a month, 14-day retention, unlimited users	Pro, $249 a month: 5 GB, 50k scores, 30-day retention	Data and scores (the free tier can also pay for overage), no seat charge
LangSmith	1 seat, 5k traces a month, 14-day retention	Plus, $39 per seat a month: 10k traces	Seats and traces; overage rate not published as text
Datadog	40k LLM spans a month, 15-day retention	$160 a month for up to 100k LLM spans	LLM call spans only, including the judge's own calls

One pricing detail to watch in LangSmith: experiments and scored traces default to extended retention, which costs more than base traces unless you turn it off. The free tiers are all large enough to find out whether evals change how your team works, which is the only question that matters in the first month.

How I rank them

Criteria first, because the ranking depends on them. I am ranking for a small team shipping an agent, where the scarce resource is engineers' attention and the goal is to know whether a change made the agent better or worse before it ships.

Braintrust. In my experience, the shortest path from "I changed a prompt" to "here are the cases that got worse." Official CI support, no seat pricing, and the experiment comparison I liked working in most.
LangSmith. On paper, the most built-in tooling for agent-specific evals: trajectories, simulated users, routing guides. Choose it first if you are on LangGraph, or if multi-turn simulation is central to what you need to test.
Datadog. The right answer if your company already runs on Datadog and wants LLM quality in the same place as latency, errors and on-call. Per-LLM-call pricing with no separate eval fee makes the bill easy to predict.

Two self-hostable options are worth knowing about even though I have not used them in production. Langfuse is MIT-licensed at its core, can be self-hosted for free, and has an official GitHub Action for experiments. It is a strong option when traces must stay on your infrastructure, as long as the judge model runs there too. Arize Phoenix is free to self-host under the Elastic License and has good tool-call and trajectory metrics. You can score production traces with its SDK, but managed continuous online evals and alerting come with Arize's commercial product.

What I would set up first

Whichever platform you pick, the order matters more than the tool:

Fifty routing cases taken from real conversations (redacted), scored by exact match, run in CI.
Three to five behavior scorers, each answering one question, with the judge prompt versioned and checked against a few dozen human labels.
Code checks on every production trace, and one judge on a small sample, scored asynchronously.
A weekly habit of turning failed production traces into new test cases.

The platform makes steps 1 to 4 faster. It cannot choose the cases for you, and the cases are where the quality comes from.

From Data Science to Kubernetes: Adeoye’s Journey Into DevOps and MLOps

Sat, 29 Aug 2026 08:42:55 +0000

From Data Science to Kubernetes: Adeoye’s Journey Into DevOps and MLOps

Developer Stories | CoderLegion

What happens when a curiosity about computers leads from geology and data science to machine learning, cloud computing, DevOps, and eventually Kubernetes?

For Adeoye, the journey has been anything but linear. His path has taken him from working with geological datasets and machine learning to exploring infrastructure, cloud platforms, containers, Kubernetes, and MLOps.

Along the way, he has learned an important lesson: building an application or model is only part of the journey. Understanding what happens underneath — how systems are deployed, connected, scaled, monitored, and recovered — can be just as important.

In this Developer Stories interview, Adeoye talks about his transition into DevOps and MLOps, his hands-on #40DaysOfKubernetes journey, troubleshooting Kubernetes failures, learning through writing, and the advice he has for developers entering DevOps or MLOps.

From Data Science to DevOps

Your profile describes you as a Data Scientist focused on DevOps and MLOps. How did you get from working with data and machine learning to becoming interested in infrastructure, DevOps and Kubernetes?

My journey into DevOps and infrastructure has been quite organic. I’ve always been fascinated by computers and the internet.

I remember seeing a computer for the first time when I was about seven years old. Watching the screen light up and respond to the actions of a senior family friend was fascinating to me. From that point on, I was always curious about what computers could do and how they worked.

As I got older, that curiosity led me into different areas of technology. At one point, I became interested in cybersecurity and ethical hacking. Like many curious teenagers experimenting with technology, I tried things like testing the security of Wi-Fi networks and eventually installed Linux on my computer.

That early exposure introduced me to a completely different side of computing and gave me my first real experience working with operating systems beyond the typical user level.

Later, when I studied Geology and Earth Sciences at university, my academic work introduced me to the world of data. Geology can be extremely data-intensive, and I found myself working with increasingly complex datasets, especially during my final-year research project.

I started with tools like Excel and eventually expanded into Power BI, Python, and R.

That experience naturally led me toward Data Science and Machine Learning. After graduation, I attended a Google Developers Bootcamp, where I learned the fundamentals of machine learning and earned a TensorFlow Developer Certificate.

At that point, I wanted to build a portfolio that demonstrated what I had learned and the projects I had worked on.

Interestingly, building that portfolio became the bridge into software development.

While trying to create and deploy websites for my projects, I started learning more about web development. I eventually built a number of websites, desktop applications, and mobile applications and experimented with hosting them on free cloud platforms.

But I became increasingly curious about how applications were actually deployed and managed in real-world cloud environments such as AWS and Google Cloud.

That curiosity led me to start learning about cloud computing and DevOps practices. I took online courses covering DevOps concepts and Google Cloud, and during that stage of my journey, I came across a free course offered by the Linux Foundation.

I applied, was accepted, and that was really where Kubernetes and infrastructure started to become a major focus for me.

As I learned more about containers, orchestration, networking, deployments, and the infrastructure behind modern applications, I realized that I genuinely enjoyed it.

Kubernetes especially changed the way I thought about software.

I had previously focused largely on building models and applications, but DevOps and infrastructure helped me understand what happens after you build something — how it is deployed, scaled, connected, monitored, and kept running.

Since then, it has been a really interesting journey.

Why MLOps?

What attracted you to MLOps specifically? What do you think developers or data scientists often underestimate about what it takes to move a machine-learning project from an experiment into something that can actually run reliably?

I became interested in MLOps during my Data Science phase.

After completing the Google Developers Bootcamp, I had a lot of ambitious ideas, one of which was to build an AI-powered seismic analyzer that could use machine learning and seismic data to identify patterns that might help with earthquake prediction.

It started as a small-scale personal project, but once I began implementing it, I quickly realized that there was much more to building an ML project than just training a model.

One of the biggest challenges I ran into was compute and the size of the datasets I would potentially need.

At the time, I was working on a Dell laptop with 16 GB of RAM, and I began to understand the practical limitations of trying to build something like that with limited resources.

I think that is something developers and data scientists often underestimate.

There is a big difference between getting a model to work in an experiment and building something that can actually run reliably in the real world.

When you're experimenting, the focus is usually on the model itself: can you train it, does it work, and how accurate is it?

But once you want to move beyond that, you start dealing with questions about where the data comes from, how it is managed, whether your work can be reproduced, and whether you have the infrastructure and compute needed to support the system.

A model might work perfectly on your local machine, but that does not necessarily mean it can be easily deployed or scaled.

You also have to think about what happens when the system fails or when the data changes over time and starts affecting the model's performance.

This realization is what really attracts me to MLOps. I am very interested in understanding what happens around the model and what it takes to move from an experiment to a useful project that can be deployed in the real world.

Learning Kubernetes by Doing

You've been documenting a hands-on #40DaysOfKubernetes journey. What made you decide to learn Kubernetes this way, and what has surprised you most along the way?

There’s a learning style that works very well for me: when I write about what I’m learning, it tends to stick.

It almost feels like I’m teaching myself the subject all over again, which is why I decided to document my Kubernetes journey.

What has surprised me most is how much I’ve learned from troubleshooting.

Sometimes I would spend days trying to fix a deployment issue, and once I finally solved it, I understood the problem so well that I knew exactly what to look for if I encountered it again.

I could also see myself getting faster with Kubernetes commands and better at diagnosing problems with each lesson.

When Kubernetes Breaks

You recently wrote about troubleshooting a real Kubernetes application failure. What did that debugging experience teach you about troubleshooting distributed systems compared with debugging a normal application?

It taught me that debugging distributed systems requires following the entire path of a request rather than focusing on one application.

In my case, the Pods were running, but small mismatches in ports, labels, and NetworkPolicies were silently breaking communication.

I learned to troubleshoot systematically, moving layer by layer from:

Service → Endpoint → Pod → Network Policy

That experience reinforced how important it is to understand the relationships between the different components rather than looking at each component in isolation.

Understanding Instead of Memorising

A lot of Kubernetes learning can become a collection of commands and YAML files. How do you approach learning the underlying concepts so that you're not simply memorising commands?

I tend to read a lot, so when I come across a Kubernetes concept I don’t understand, I take the time to dig into it through articles, documentation, or YouTube videos that explain it in detail.

I try not to just learn the command that fixes a problem, but understand why it works.

So, with every problem I encounter and every concept I initially struggle with, I come away with a stronger understanding of how Kubernetes actually works.

The Hard Parts of Kubernetes

You've been working through topics such as Ingress, ConfigMaps, Secrets, resource requests and limits, HPA/VPA, and etcd backup and recovery. Which Kubernetes concept has been the most difficult to truly understand, and why?

The concept I found most difficult to truly understand was Service Accounts and authentication, especially the certificates and tokens involved.

At first, I struggled to understand how Pods authenticate with the Kubernetes API and how that differs from authorization and RBAC.

Working through Service Accounts and CA certificates, and how they are mounted into Pods, helped me understand that authentication is about proving who you are, while authorization determines what you are allowed to do.

That distinction was one of the biggest hurdles for me.

Why Backups and Recovery Matter

You recently worked through etcd backup and restore. Why do you think developers often pay more attention to making systems work than to proving that they can actually recover when something goes wrong?

I think it often comes from the assumption that if you’ve built something correctly, it won’t fail.

Confidence in your abilities is important, but errors are inevitable.

Even machines fail, so systems need to be designed with the assumption that something will eventually go wrong.

Learning in Public

You regularly share your progress and technical discoveries publicly. What have you learned from explaining your work to other developers, and has writing about something ever exposed gaps in your own understanding?

Like I mentioned earlier, my learning style is such that when I write about something, it tends to stick more, and it also helps me identify gaps or areas where I might be wrong.

I remember while learning CS50 from Harvard, the instructor explained that one of the best ways to understand something is to explain it to a rubber duck.

A rubber duck is a complete beginner, so you have to break the concept down as simply as possible.

That is what I’ve been trying to do with my writing: break technical concepts down to the point where almost anyone can understand them.

The Developer You’re Becoming

How has your approach to technology changed since you started? Are you now more interested in understanding how systems work underneath rather than simply getting an application running?

To be honest, I’m much more interested in understanding how a system works underneath now.

Earlier in my journey, my approach was more about making sure the application worked and didn’t break during testing, even if I didn’t fully understand what was happening underneath or why something was breaking.

My mindset has shifted quite a bit since then.

Now I want to understand what makes the system work in the first place, so that when something does break, I can trace the problem back to the specific component that caused it rather than simply trying to find a quick fix.

Advice for Developers Entering DevOps or MLOps

For someone coming from software development, data science, or machine learning who wants to move into DevOps or MLOps, what would you recommend they learn first — and what mistakes should they avoid?

I would say start with Linux, networking, and Docker.

These give you a good understanding of what is happening underneath before moving into tools like Kubernetes.

I’d also say don’t focus too much on memorising commands.

Learn how to troubleshoot and understand why things work or break, because that knowledge will follow you across different tools and technologies.

Final Thoughts

Adeoye’s journey is a good reminder that careers in technology don't always follow a straight path.

A background in Geology led to working with data. Data Science led to Machine Learning. Building ML projects created an interest in software development, which eventually led to cloud computing, DevOps, MLOps, and Kubernetes.

And perhaps the biggest lesson from his journey is that understanding why something works can be far more valuable than simply knowing the command that makes it work.

His #40DaysOfKubernetes journey also highlights another powerful approach to learning: build, troubleshoot, explain what you learned, and repeat.

That process doesn't just create technical knowledge — it helps develop the ability to understand complex systems from the inside out.

MLOps for Engineers: How to Take an AI Model from Jupyter to Production Without Dying in the Attempt

Thu, 27 Aug 2026 19:08:27 +0000

There is a statistic from Gartner that every data team knows and most prefer to ignore: 87% of Machine Learning and AI projects never make it to production. They remain trapped in "prototype limbo": a Jupyter notebook on a data scientist's laptop boasting an impressive 96% accuracy over a static CSV file from 2023, but which no one knows how to deploy, version, update, or monitor within an enterprise's live infrastructure.

The reason for this widespread failure is neither mathematical nor algorithmic. It's not a matter of needing more hyperparameters to tune or deeper neural network layers to stack. It is a software engineering and operations failure.

As demonstrated in Google's landmark paper "Hidden Technical Debt in Machine Learning Systems" (Sculley et al.), the actual ML model code accounts for a mere 5% to 10% of the entire production system. The remaining 90% is infrastructure: data extraction and verification, dependency management, automated retraining, artifact versioning, observability against the dreaded Data Drift, and continuous governance.

This integrative discipline is called MLOps (Machine Learning Operations). And after building, deploying, and operating real production pipelines on this blog — from demand forecasting with Prophet in the S&OP series to autonomous agents in the Autopilot series and the Obsolescence Radar —, this article is the practical guide that distills the journey from a sandbox script to an industrial production system.

Anatomy of the Problem: Why Traditional Software Engineering Falls Short with ML

In traditional software development (DevOps), system behavior depends entirely on source code. If you write a deterministic function, test it with unit tests, and deploy it via CI/CD, the system behaves predictably as long as the underlying infrastructure remains healthy.

In Machine Learning and generative AI systems, behavior depends upon an interdependent trinity: Code + Data + Model.

Code may remain unchanged, yet if the statistical distribution of real-world data shifts (which happens constantly across supply chains, financial markets, and customer behaviors), model performance degrades silently.
Reproducibility is non-trivial: retraining the exact same Python script with today's data produces a completely different binary artifact than last week's run.
Failures don't throw an HTTP 500 error: a production model doesn't crash like a web server; it simply begins serving garbage predictions with 99% statistical confidence.

To keep your system from becoming an uncontrollable black box, your architecture must stand upon four fundamental pillars.

Pillar 1: Comprehensive Versioning (Code, Data, and Models)

If you cannot recreate the exact environment, data snapshot, and parameters that produced a prediction six months ago, your system is neither reproducible nor auditable. In regulated enterprise environments, this is not merely good engineering hygiene; it is a strict legal requirement under the EU AI Act (Article 12 on automated logging and traceability).

Versioning in MLOps spans three distinct layers:

Code Versioning: Standard Git. Centralized repository with protected branches, pull request reviews, and version tags.
Data Versioning (DVC / Delta Lake): Git was never designed to store gigabyte- or terabyte-scale binary files. Tools like DVC (Data Version Control) create lightweight metadata pointer files versioned in Git, while actual datasets reside in object storage (Amazon S3, Google Cloud Storage, or Supabase Storage). This allows a simple git checkout v1.2.0 to restore both the training code and the exact data snapshot that fed it.
Model Registry (MLflow / Weights & Biases): A centralized catalog acting as the "Docker Hub" for trained models. Each registered model tracks serialized weights (ONNX, Pickle, Safetensors), hyperparameters, validation metrics, author, associated Git commit hash, and lifecycle stage (Staging, Production, Archived).

Pillar 2: Experiment Tracking and Pipeline Automation

A data scientist's or ML engineer's workflow is inherently iterative. Testing dozens of feature permutations, model algorithms, Pandas transformations, and regularization parameters without systematic logging leads straight to chaos: scattered notebook files, models named final_model_v2_really_final.pkl, and complete loss of institutional knowledge.

MLflow and Weights & Biases (W&B) solve this by systematically intercepting every training run:

import mlflow
import mlflow.prophet
from prophet import Prophet

# Start experiment tracking
mlflow.set_experiment("sop_demand_forecasting")

with mlflow.start_run(run_name="prophet_multiplicative_v3"):
    # Log configuration parameters
    params = {
        "seasonality_mode": "multiplicative",
        "changepoint_prior_scale": 0.05,
        "n_changepoints": 20
    }
    mlflow.log_params(params)
    
    # Train the model
    model = Prophet(**params)
    model.fit(train_df)
    
    # Evaluate and log performance metrics
    metrics = evaluate_forecast(model, test_df)
    mlflow.log_metrics({
        "mape": metrics["mape"],
        "rmse": metrics["rmse"],
        "coverage_p95": metrics["coverage"]
    })
    
    # Automatically register the artifact
    mlflow.prophet.log_model(model, artifact_path="prophet_model")

By structuring experiments with automated tracking, comparing 50 model iterations transforms from a guessing game into an instant analytical query on a unified dashboard.

Pillar 3: CI/CD and Automated Deployment (CT: Continuous Training)

In MLOps, CI/CD expands to incorporate a vital third dimension: Continuous Training (CT).

CI (Continuous Integration): Goes beyond validating Python syntax and running unit tests (pytest, flake8). It executes specialized data validation tests: verifying that incoming schemas match expectations, null values remain within acceptable tolerances, and feature ranges stay valid (using libraries like Great Expectations or Pydantic).
CD (Continuous Delivery / Deployment): Packages the validated model into an optimized Docker container or a FastAPI microservice (as showcased in Observability Part 6) and automatically deploys it upon passing regression test suites.
CT (Continuous Training): When production monitoring detects performance degradation or new data batches arrive, an automated pipeline (orchestrated via GitHub Actions, Prefect, or Airflow) retrains the model, verifies that new metrics surpass the baseline champion model, and automatically promotes the candidate to the Model Registry.

Pillar 4: Production Observability (The War Against Drift)

Once your model is deployed and serving batch or real-time inferences, the real challenge begins. Models degrade over time due to two distinct mathematical phenomena:

1. Data Drift (Covariate Shift)

The statistical distribution of input variables ($P(X)$) changes relative to the training distribution, even if the underlying relationship between inputs and outputs holds steady.

Real-world example: A demand forecasting model trained prior to a global supply chain disruption experiences a sudden surge in supplier lead times. The model receives valid numbers, but operates in a vector space region where it was never trained.

2. Concept Drift

The mathematical relationship between input features and target variables ($P(Y|X)$) shifts over time.

Real-world example: In customer purchasing predictions, a product historically bought primarily during winter becomes an all-season hit due to a social media trend. The inputs remain identical, but consumer behavior has fundamentally evolved.

# Data Drift detection example using Evidently AI
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

data_drift_report = Report(metrics=[DataDriftPreset()])
data_drift_report.run(reference_data=reference_df, current_data=production_df)

# If drift exceeds threshold, trigger automatic alerting & retraining
if data_drift_report.as_dict()["metrics"][0]["result"]["dataset_drift"]:
    trigger_mlops_alert("DATA_DRIFT_DETECTED: Triggering automated retraining pipeline")

Tools like Evidently AI or Prometheus + Grafana allow data engineering teams to track statistical drift tests (such as Kolmogorov-Smirnov for numerical features or Chi-Square tests for categorical data) and alert engineers long before degraded accuracy causes financial losses.

Real Production Case: Moving Prophet from Notebook to S&OP Pipeline

To see how MLOps applies in real industrial contexts, consider the evolution of our demand planning pipeline in the S&OP series:

Stage	Initial Sandbox (Jupyter)	Production Architecture (MLOps)
Data Ingestion	Manual loading of static `sales_2023.csv`	Automated Supabase PostgreSQL sync with schema assertion
Data Hygiene	Manual `df.dropna()` without validation	Automated Z-Score outlier detection as detailed in Data Hygiene
Training	Manual top-to-bottom cell execution	Decoupled pipeline running weekly via GitHub Actions
Bayesian Inference	Single point forecast	Probabilistic confidence intervals directly driving Safety Stock calculations
Consumption	Static `plt.show()` inline charts	FastAPI microservice feeding the PuLP linear optimization engine (Part 3)
Monitoring	None	Continuous weekly MAPE tracking triggering automated retraining

This engineering transition transformed an isolated data analysis script into an automated, resilient, enterprise-grade system.

From MLOps to LLMOps: The Agentic Frontier

In 2026, the rise of LLMs and autonomous agent architectures hasn't eliminated the need for MLOps; it has evolved it into LLMOps.

When designing multi-agent workflows like those in the Autopilot series or navigating Fine-Tuning vs Prompt Engineering vs RAG, operational challenges take on fresh complexities:

Automated Prompt CI/CD Regression Testing: Adjusting an agent's system prompt can fix one edge case while subtly breaking three others. LLMOps pipelines run automated regression test suites against curated evaluation datasets using frameworks like RAGAS or DeepEval.
Token Cost and Latency Observability: As discussed in The Hidden Economics of AI, token consumption and inference latency are mission-critical operational metrics that must be monitored just as closely as server CPU or memory utilization.
Proactive Security Governance: Given vulnerabilities like Prompt Injection, LLMOps observability must continuously audit tool execution payloads (Tool Calling / MCP) and flag anomalous behaviors before they trigger unauthorized downstream actions.

The EU AI Act Connection: Compliance as Code

The EU AI Act has elevated MLOps from a best-practice engineering recommendation to a mandatory legal requirement:

Article 9 (Risk Management System): Mandates continuous risk identification and mitigation across the entire AI lifecycle (fulfilled via automated CI/CD and continuous testing).
Article 12 (Record-Keeping & Logging): Requires automated event logging throughout production operations (fulfilled via Model Registries and observability tracking).
Article 15 (Accuracy, Robustness & Cybersecurity): Demands that high-risk systems maintain consistent accuracy benchmarks and resist adversarial manipulation.

Implementing rigorous MLOps practices allows companies to satisfy these regulatory mandates natively through infrastructure-as-code, eliminating the friction and overhead of retrospective audits.

Conclusion: MLOps is the Maturity of Artificial Intelligence

Anyone can clone a repository, open a Jupyter notebook, and fit a model in three lines of Python. But real engineering is not about making an algorithm work once in a sanitized sandbox; it is about ensuring it runs reliably, accurately, and securely ten thousand times a day in production.

The continuous MLOps loop — design, train, validate, deploy, monitor, and retrain — is the purest technological expression of the PDCA (Plan-Do-Check-Adjust) cycle that W. Edwards Deming pioneered for industrial quality excellence.

If you aspire to build AI systems that generate enduring enterprise value, leave behind the comfort of the solitary notebook and embrace the discipline of MLOps. Your future self — and your on-call engineering team — will thank you forever.

Sources of Interest:

AI is a Part of Business Now

Wed, 05 Aug 2026 03:50:04 +0000

Live: https://b2b-buying-signal-detector-ay7hvbk34hwpmr899syjqh.streamlit.app/

GitHub: https://github.com/Danish08654/B2B-Buying-Signal-Detector

About

An AI-powered intent detection engine that analyses company signals across different field and engagement to predict which companies are in an active buying.

🔬 AI Multi-Agent Research System

Fri, 31 Jul 2026 14:25:34 +0000

Instead of relying on a single LLM call, I designed a multi-agent system where different agents collaborate like a real research team.

System Architecture:

1) Planner → defines research strategy & search queries
2) Searcher → gathers relevant information (offline/online mode)
3) Analyst → extracts insights, patterns, and key findings
4) Writer → generates a structured research report
5) Fact-Checker → verifies and refines final output

Live: https://multi-ai-agent-research-system-mgcp4yp8nekmdeznnf5szg.streamlit.app/

How a Misconfigured AI Pipeline Can Take Down an Ecosystem

Thu, 23 Jul 2026 10:00:49 +0000

When a major AI lab's infrastructure accidentally hammers a shared open-source platform with millions of requests, it stops being a theoretical risk - it's a production incident.

The Hidden Blast Radius of Poorly Throttled AI Pipelines

Large AI systems frequently pull from external model hubs - places like Hugging Face - to download weights, tokenizer configs, or dataset files. When an automated training or evaluation pipeline runs without rate limiting or caching, it can generate enormous request volumes that look indistinguishable from a DDoS attack to the receiving server.

The pattern is straightforward: a job scheduler kicks off hundreds of parallel training runs, each worker independently fetches the same remote artifact, and suddenly a single repository is fielding tens of thousands of concurrent GET requests. No malicious intent required. Just missing guardrails.

This matters beyond the drama of who did it. Any team running distributed fine-tuning - the process of adapting a pre-trained model on your own data - or large-scale evaluation pipelines is one misconfiguration away from causing the same kind of collateral damage. The open-source model ecosystem runs on shared infrastructure that was never designed for the throughput demands of modern AI training clusters.

Real Example

Here's the type of pattern that causes this - fetching a model file inside a parallelized worker with no cache or retry discipline:

# Problematic: every worker independently fetches from remote
from transformers import AutoModel

def train_worker(config):
 model = AutoModel.from_pretrained("org/model-name") # no cache, no lock
 # ... training logic

A safer pattern uses a local cache directory and pre-fetches once before workers spin up:

from transformers import AutoModel
import os

# Pre-fetch once, workers read from disk
os.environ["TRANSFORMERS_CACHE"] = "/shared/cache"
model = AutoModel.from_pretrained("org/model-name", local_files_only=False)

# Workers downstream use local_files_only=True
def train_worker(config):
 model = AutoModel.from_pretrained("org/model-name", local_files_only=True)

Adding local_files_only=True in worker processes means they'll fail fast if the artifact isn't cached rather than hammering a remote server. Pair this with a artifact pre-download step in your pipeline orchestrator - whether that's Airflow, Prefect, or a shell script - before any parallel workers launch.

Key Takeaways

Distributed AI pipelines can unintentionally generate DDoS-level traffic against shared infrastructure like model hubs, with no bad intent required
Setting TRANSFORMERS_CACHE to a shared volume and separating download from training steps eliminates redundant remote fetches across parallel workers
Rate limiting and local_files_only=True in worker code are low-effort guardrails that protect both your pipeline reliability and the ecosystem you depend on

The shared infrastructure that makes open-source AI accessible is fragile in ways most pipeline builders never consider until something breaks publicly.

If you run distributed training or eval jobs, have you explicitly tested what happens to your external fetch calls when 50 workers start simultaneously?

Sources referenced: HackerNews discussion - "OpenAI's accidental attack against Hugging Face is science fiction that happened"

If you're just getting started, think of AWS, GCP, and Azure as cloud platforms

Sat, 11 Jul 2026 11:52:48 +0000

What is Cloud Computing?

Traditionally, if you wanted to run an application, you had to buy:

Physical servers
Storage devices
Networking equipment
Cooling systems
Backup power
Security infrastructure

Cloud providers own this infrastructure and let you use it on demand.

The Three Major Cloud Providers

                    **1. Amazon Web Services (AWS)

Released: 2006

Company: Amazon Web Services

AWS is the world's largest cloud platform.

                   ****Common Services

EC2 (Virtual Servers)
S3 (Object Storage)
RDS (Managed Databases)
Lambda (Serverless Functions)
IAM (Identity & Access Management)
CloudWatch (Monitoring)
SageMaker (Machine Learning)
Bedrock (Generative AI)
```
                   **Used by
```
Netflix
Airbnb
NASA
Samsung
BMW
```
                 **Strengths
```
Largest service ecosystem
Very mature platform
Excellent scalability

Strong AI and ML offerings

         **2. Google Cloud Platform (GCP)**

Released: 2008
Company: Google Cloud
Google Cloud is known for AI, machine learning, big data, and Kubernetes.

                 **Common Services

Compute Engine
Cloud Storage
Cloud SQL
BigQuery
Cloud Run
Vertex AI
Kubernetes Engine (GKE)
```
                **Used by
```
Spotify
PayPal
Snapchat
Twitter/X
```
              **Strengths
```
Excellent AI ecosystem
Big data analytics
High-performance networking
```
        **3. Microsoft Azure
```

Released: 2010
Company: Microsoft Azure
Azure is widely adopted by enterprises and organizations already using Microsoft technologies.

             **Common Services

Virtual Machines
Blob Storage
Azure SQL Database
Azure Kubernetes Service (AKS)
Azure OpenAI Service
Azure AI Search
```
             **Used by
```
Walmart
Adobe
LinkedIn
Many government organizations
Large enterprise businesses
```
            **Strengths
```
Enterprise integration
Strong business adoption

Everpure Data Stream Takes Aim at the AI Pipeline Problem

Wed, 17 Jun 2026 14:15:00 +0000

Enterprises aren't running out of data. They're running out of ways to make it useful.

The gap between raw enterprise data and production-ready AI is where most projects die. Data scientists spend months wrangling ingestion pipelines, cleaning unstructured content, building vector embeddings, and trying to keep everything in sync as source data changes. According to a June 2026 IDC survey of more than 1,300 organizations, 34% of companies report their data scientists spend more than half their time waiting on IT — waiting for datasets, waiting for GPU time, waiting for requests to be fulfilled — rather than actually building models.

At Pure Accelerate 2026, Everpure announced Everpure Data Stream, a capability designed to collapse that pipeline work from months to days. It's the company's most direct answer yet to what Rob Lee, Chief Technology and Growth Officer at Everpure, described during the press conference as the defining bottleneck of enterprise AI: not compute, not software, but data.

The Pipeline Problem

To understand what Data Stream addresses, it helps to understand what AI-ready data actually requires.

Data doesn't arrive AI-ready. It arrives as PDFs, Word documents, PowerPoint files, images, database records, and unstructured content scattered across on-premises systems, SaaS applications, cloud storage, and legacy environments. Before any of that can feed an AI model, it has to be ingested, curated, classified, embedded into vector representations, indexed for retrieval, and connected to a generation layer. That full sequence — Ingest, Curate, Classify, Embed, Index, Retrieve, Generate — is what Data Stream automates end to end.

The alternative is what most enterprises are doing today: armies of engineers building and maintaining custom pipelines, manually handling data preparation, and rebuilding that work every time source systems change or model requirements shift. The IDC research puts the cost of that approach in concrete terms: better workload scheduling (49%) and faster network data delivery (48%) are the top two improvements IT leaders say they need to improve GPU utilization. Both point directly to the data pipeline as the bottleneck starving compute clusters.

Idle GPUs are expensive. As Sabur Mian, CEO and Founder of STN, put it: "Idle GPUs are economically destructive."

What Data Stream Does

Data Stream brings AI capabilities directly to enterprise data where it already lives. It doesn't require moving data into a new system or copying it into a vendor-controlled environment. It runs on infrastructure the customer owns, meaning content never leaves their environment — a critical requirement for enterprises with data sovereignty, security, or compliance constraints.

The pipeline it automates handles three core jobs.

Converting unstructured data for AI use. Most enterprise data isn't structured. PDFs, documents, presentations, and images contain valuable information that AI models can't directly consume. Data Stream reads the formats data actually lives in and converts it into representations AI can work with — without manual intervention.

Maintaining strict governance controls. Stream-level access controls keep information within the corporate network. As AI agents increasingly need to access sensitive data to complete tasks — compensation records, financial data, personally identifiable information — the governance model has to be embedded in the pipeline itself, not bolted on afterward. Data Stream enforces those controls at the point of data flow.

Scaling storage and compute independently. AI workload requirements change. A model that works fine at one scale may need dramatically different infrastructure three months later. Data Stream's architecture lets storage and compute scale independently to match evolving model requirements, avoiding the permanent over-provisioning that fixed infrastructure typically demands.

The NVIDIA Partnership

Data Stream extends the NVIDIA AI Data Platform reference design, giving it a GPU-accelerated pipeline from ingestion through inference. The integration replaces manual data ingestion and manipulation with automated processing that gets from raw enterprise data to real-time AI results significantly faster.

Everpure is also developing next-generation AI solutions with NVIDIA STX — a modular foundation for AI-native storage using NVIDIA Vera and the NVIDIA BlueField-4 STX storage processor. The collaboration is focused on bringing acceleration, security, and intelligent data services closer to enterprise data as organizations scale agentic AI deployments. Timeline details weren't disclosed, but the direction is toward tighter integration between the storage layer and GPU infrastructure at the hardware level.

Jason Hardy, VP of Storage Technology at NVIDIA, described the goal: organizations need infrastructure that bridges secure, governed enterprise data with accelerated computing, allowing them to scale from AI experimentation to full production intelligence.

FlashBlade as the Performance Foundation

Data Stream runs on FlashBlade, Everpure's scale-out file and object storage platform. The performance numbers matter here because data pipeline throughput directly determines how fast AI models can train and how responsive inference workloads can be.

FlashBlade//EXA scales to 800 Gbps of throughput per node while maintaining low latency — the combination required to keep GPU clusters fed at line speed. Organizations start with FlashBlade//S and scale to FlashBlade//EXA as AI factories grow, with Everpure's Evergreen architecture enabling non-disruptive upgrades throughout.

Portworx, Everpure's container platform, handles deployment and orchestration across the pipeline — managing AI workloads from edge to core data center and providing the data portability layer that lets containerized AI applications move across infrastructure without losing access to their data.

Two customer deployments illustrate what that infrastructure combination produces in practice.

STN, a GPU cloud provider, standardized on Everpure FlashBlade after a proof of concept and had an end customer live within a week of completing migration. FlashBlade//EXA allows them to scale thousands of GPUs with consistent throughput, driving up to 20% performance improvement for customers.

Beyond, a private AI factory in Central and Eastern Europe, deployed 3 petabytes of FlashBlade//S alongside NVIDIA DGX SuperPOD infrastructure. The result: 80% faster AI workloads, a 1.2 PUE for energy efficiency against a regional average of 1.5, and on-demand scaling through Evergreen//One.

Where This Fits in the Larger Architecture

Data Stream sits within Everpure's Unified Data Plane — the shared storage foundation that spans cloud, core, and edge across every workload type. It's the execution layer for making data AI-ready, sitting alongside Everpure Data Intelligence, which handles the discovery, classification, and contextualization work upstream.

The relationship between the two is straightforward. Data Intelligence figures out what data you have, where it lives, what it means, and how it connects to everything else in the enterprise. Data Stream takes that understood, classified data and converts it into the pipeline format AI applications and agents actually need — vectorized, indexed, and ready for retrieval.

Together they form what Everpure describes as the AI-ready data path: find and catalog, classify and contextualize, optimize for AI, govern and secure, execute at scale.

For engineering teams, the practical implication is that neither product alone solves the problem. Data Intelligence without Data Stream gives you a semantics layer but no automated pipeline to AI consumption. Data Stream without Data Intelligence gives you pipeline automation but without the context that makes retrieval accurate and agents reliable.

The 94% Problem

The IDC survey found that 94% of respondents consider data quality important or very important to AI project success. The top contributors to poor data quality: redundant data, multiple storage silos, and obsolete data. All three are symptoms of the same underlying condition — data pipelines that are manual, fragmented, and out of sync with source systems.

Data Stream's bet is that automating the pipeline end to end, running it on infrastructure the organization controls, and keeping it synchronized with live enterprise data addresses those quality problems at the source rather than compensating for them downstream.

Sixty percent of organizations in the IDC survey said their storage infrastructure needs significant improvement or a total refresh to support AI workloads. For organizations actively evaluating AI infrastructure, Data Stream is Everpure's argument that the storage layer — not just the model layer — is where that investment needs to happen.

The full pipeline from raw enterprise data to production AI results is available now. Organizations that have been rebuilding custom ingestion work for each new AI project have an alternative worth evaluating.

Everpure (NYSE: P) provided press access to Pure Accelerate 2026. The IDC research cited in this article was commissioned by Everpure.

MLOps Lifecycle: Stages, Workflow, and Best Practices

Tue, 02 Jun 2026 17:13:11 +0000

A machine learning model that performs well on day one will not remain stable by default. Performance can degrade over time due to data drift, changes in user behavior, evolving feature sets, or updates to upstream systems. These changes rarely cause immediate failure, but they reduce reliability and make model behavior harder to understand.

The core issue is not model quality, but a lack of coordination across the lifecycle. Decisions made early in the lifecycle affect every stage that follows. When stages operate in isolation, traceability breaks down. For example, code versioning may capture model changes, but not dataset lineage, feature definitions, or runtime behavior.

MLOps addresses this by treating machine learning as a continuous, end-to-end lifecycle. It connects data, features, training, deployment, monitoring, and governance into a single operating model. Each stage introduces its own assumptions and dependencies, from training and validation to deployment, monitoring, and governance.

Summary of key MLOps lifecycle concepts

Stage	Activities and Outputs
Data Ingestion and Labeling	Collect raw data (logs, databases, APIs, and sensors), annotate or label it if necessary, and clean it. The output will be versioned datasets or snapshots.
Feature Engineering	Take raw data and transform it into features (e.g., normalization, encoding, and aggregation) and register these features in a feature store.
Model Training and Experimentation	Perform training jobs and hyperparameter tuning. The output of this stage will be trained model artifacts like weights and checkpoints.
Validation and Testing	Test new models against holdout or test data. The output will be accuracy, loss, fairness metrics, and validation reports.
Packaging and CI/CD	Package the model into a deployable artifact or container and push it to a model registry or a container registry.
Deployment and Rollout	Deploy the model to production (REST endpoint, batch service, etc.). Manage traffic with canary releases and/or blue-green deployments. For LLM applications, Configs extends these capabilities to prompt versioning and model provider management.
Monitoring and Observability	Monitor system health: latency, error rates, etc. Monitor machine learning health, including elements like prediction quality and data drift.
Feedback and Retraining	Collect new labeled data and initiate the process of retraining the model. Schedule retraining runs using the newly collected data.
Governance and Approval	Conduct human-in-the-loop reviews and compliance checks before deploying the model. Maintain documentation of the models (e.g., model cards and data sheets), and implement automated policy checks.

The following diagram shows how the major MLOps lifecycle stages connect in practice, from data ingestion through deployment, monitoring, and retraining, along with the operational outputs produced at each step.

Data ingestion and preparation

Data as a first-class production artifact

Most ML systems do not make data ingestion a control boundary, instead treating it as a background process. Initially, everything looks good, but then issues creep in, such as missing columns, silent null propagation, schema changes, late arrival of upstream data, or unknown outliers. There is no catastrophic failure, just gradual degradation of model performance, making it hard to debug and identify exactly what original data was used.

Data ingestion should be a first-class citizen in the MLOps workflow. It is essential to establish reproducibility, compliance, and reliability for models. Determinism and measurable data quality should be achieved.

Ingestion as a control layer

Data ingestion must control the entry of all data being routed and validated. Data should be collected either in batches or streams before undergoing deterministic data cleansing transformations. Before any data is saved, schema requirements should be validated. In addition, each ingestion point should create a snapshot or version for future reference. Data lineage and quality metrics should be recorded at every stage along the processing route, so if validation fails, training on that data stops completely.

In MLOps, one key operational choice is whether a system should be fail-closed or fail-open. Fail-closed systems stop processing as soon as an anomaly is detected, maximizing safety. Fail-open systems continue processing with fallback logic, maximizing availability. The decision should depend on business risk, not the default implementation.

The pseudocode below shows a simplified ingestion control flow: load raw data, validate its schema, apply deterministic transformations, measure drift, and then store the resulting dataset version and metadata for downstream training.

raw_data = load_from_source(config["data"]["source"])

validate_schema(raw_data, config["data"]["schema"])

cleaned = apply_transformations(
    raw_data,
    null_strategy=config["data"]["null_handling"],
    outlier_strategy=config["data"]["outlier_policy"],
)

drift_score = compute_drift(cleaned)

if drift_score > config["data"]["drift_threshold"]:
    alert("Distribution shift detected")

dataset_version = snapshot_dataset(cleaned)
store_metadata(dataset_version, drift_score)

For high-risk ML workflows such as regulated decisions, fraud detection, or safety-sensitive systems, ingestion pipelines should usually fail closed. In lower-risk cases, teams may choose fail-open behavior with explicit fallback logic, but that should be a conscious business decision rather than an implicit default.

Deterministic validation signals

Deterministic validation means data checks that always produce the same pass/fail outcome for the same data based on predefined rules. If a required column disappears, a null rate exceeds an allowed threshold, or a distribution shift crosses a defined limit, the pipeline should respond predictably every time. These checks are often the first reliable sign of upstream data problems, such as schema changes, silent null propagation, or newly introduced categorical values.

In addition to checking whether columns exist, validating data effectively should include the following aspects:

Determining null counts and validating other attribute values
Validating that attribute values fall into the correct range
Limiting the number of categories available for categorical attributes
Measuring distributional shifts in an attribute through either a PSI or KS test
Measuring the number of duplicate records before any data goes into your model at all

Operational validation heuristics

In practice, ingestion validation is implemented as a set of operational heuristics that help teams interpret failures quickly. The signal itself matters, but so does what it usually implies operationally, because that determines whether the right response is to stop the pipeline, investigate upstream systems, or trigger a fallback path.

Signal	Interpretation
Missing required column	Usually indicates that an upstream schema or API contract changed and downstream transformations may no longer be valid
Null rate > threshold	Often suggests corrupted source records, partial extraction failures, or broken joins in the upstream pipeline
Distribution drift > threshold	May indicate a change in user behavior, source population, collection logic, or rollout conditions
High duplicate rate	Often points to replayed ingestion jobs, duplicate event delivery, or broken deduplication logic
Unseen categories	Can break encoders or produce invalid feature mappings if serving logic was built against a fixed category set

Data versioning and lineage

Having immutable dataset snapshots is critically important to ensure reproducible results. To allow reproducible training runs, each training run must reference the dataset version ID, schema hash, transformation configuration, and associated quality metrics. Without versioning, retraining becomes non-deterministic.

In regulated environments, ingestion needs to automatically enforce PII masking, field-level anonymization, and retention tagging. These controls should be enforced automatically as part of the ingestion pipeline rather than handled through ad hoc manual review because manual compliance steps are hard to audit and easy to bypass under delivery pressure.

Configuration and feature flag controls

In mature ML systems, ingestion rules should be controlled through external configuration rather than hard-coded into pipeline logic. This allows teams to adjust schema strictness, null-handling rules, drift thresholds, and anonymization behavior without redeploying the pipeline. The YAML below shows one way to define those ingestion policies declaratively.

data:
  source: "s3://raw/customer_data"
  schema: "schemas/customer_v3.yaml"
  null_handling: "impute_median"
  outlier_policy: "clip_99_percentile"
  drift_threshold: 0.1

validation:
  enforce_strict_schema: true
  max_null_rate: 0.05

Feature flags can control behaviors such as strict schema validation, drift blocking, and auto-anonymization. This enables the gradual introduction of more stringent validation, with instant rollback if the rules block production unexpectedly.

Ingestion-level operating metrics

The ingestion stage should expose a small set of operating metrics so teams can tell whether data is arriving on time, passing validation, and staying within expected quality bounds. These are stage-specific signals used to manage data intake, not a replacement for the broader production monitoring discussed later in the article.

Data intake needs to be measurable.

Key metrics:

Batch success rate
Ingestion latency
Drift score per batch
Null rate per critical feature
Rejected batch percentage
Schema violation count

Because ingestion is the first control boundary in the lifecycle, failures and drift detected here often surface before model-level symptoms appear in production. When ingestion is declarative, versioned, validated, and measurable, downstream training and deployment become far more reproducible.

Outputs

Versioned dataset snapshots
Validation reports and schema versions
Recorded data quality metrics
Metadata required for reproducibility

Feature engineering

Feature engineering is the lifecycle stage where raw, validated data is converted into the model inputs used during training and inference. In MLOps, this stage matters because feature definitions must remain consistent across offline training and online serving. If the transformation logic differs between those environments, the model may behave well in evaluation but degrade in production due to training-serving skew.

Defining the feature contract before transformation

With robust ML systems, feature definitions serve as the single source of truth; the transformation code simply implements them. The use of a feature-first approach helps make transformations deterministic, reducing the risk of training-serving skew. This consistency must extend across both offline feature stores used for training and backtesting, and online feature stores used for real-time inference. Aligning these environments helps prevent silent feature drift, invalid values, or data corruption in production.

Deterministic feature transformations

Feature transformations should be deterministic: The same input should produce the same output when the same feature definition and configuration are applied. This is what allows training, backtesting, and live inference to remain aligned. Tools such as Pandas, Spark, or feature platforms such as Feast can be used to implement that logic.

import pandas as pd
from sklearn.preprocessing import OneHotEncoder, StandardScaler

# Example: Scaling numeric features
scaler = StandardScaler()
scaled_features = scaler.fit_transform(df[["age", "income"]])

# Example: Encoding categorical features
try:
    encoder = OneHotEncoder(sparse_output=False)
except TypeError:
    # scikit-learn < 1.2 uses the sparse parameter name.
    encoder = OneHotEncoder(sparse=False)

encoded_features = encoder.fit_transform(df[["gender", "region"]])

Unit tests and train-serving consistency

Unit tests help verify both transformation correctness and train-serving consistency. In practice, that means confirming that the same feature logic used during training is also used when live requests are processed in production.

import pandas as pd
from sklearn.preprocessing import StandardScaler


def test_feature_scaling():
    df_test = pd.DataFrame({"age": [20, 40]})
    scaler = StandardScaler().fit(df_test)
    transformed = scaler.transform(df_test)

    assert transformed[0][0] < transformed[1][0]  # Scaling check

Ensure that the same transformation logic is applied during both training and serving to prevent training-serving skew. Automate feature value validation before training, which can include range and null checks.

Monitoring feature distributions

Teams usually encode feature-level validation rules separately from transformation code so they can check whether important features remain within expected bounds over time. The example below shows a simple configuration for monitoring a few feature ranges.

feature_monitoring:
  features:
    - age
    - income
    - purchase_count
  validations:
    age: [0, 120]
    income: [0, 1000000]
    purchase_count: [0, 1000]

Feature registry and versioning

Store feature definitions and pipelines in a feature registry to ensure consistency.

{
  "feature_set": "customer_features",
  "version": "v1",
  "features": ["age", "income", "purchase_count"],
  "validation_status": "passed"
}

Use Git or a feature registry to track all changes. Versioned feature pipelines support reproducibility across both training and production.

Outputs

Feature transformation pipelines
Generated feature tables or vectors
Versioned feature definitions in a registry

Model training and experimentation

Once feature sets are available, the next stage is to train candidate models and record the context needed to reproduce and compare those runs later.

Careful automation of training and experiment tracking helps improve reproducibility, consistency, and the ability to compare different models with each other at different times.

Automating model training

Whenever possible, the training process should be automated. This includes scheduling regular training runs, running hyperparameter sweeps, and retraining models when new data becomes available. Automated pipelines save time and reduce human error, especially when managing multiple models or experimenting with different parameters.

Tracking experiments

Every model training run should be tracked to ensure reproducibility and facilitate later comparisons. This means logging the hyperparameters used, such as learning rate and number of trees, dataset snapshots, code versions, and training and validation metrics.

For example, this can be done using MLflow in Python:

import mlflow

with mlflow.start_run():
    mlflow.log_param("learning_rate", 0.01)
    mlflow.log_param("num_trees", 100)

    # Training code goes here
    model.fit(X_train, y_train)

    # Log evaluation metrics
    accuracy = model.score(X_val, y_val)
    mlflow.log_metric("val_accuracy", accuracy)

    # Save the trained model
    mlflow.log_artifact("model.pkl")

This method tracks all of an experiment, and you can repeat the model or compare it with any other run later.

Controls and best practices

To prevent problems during training, configure an early stopping rule and define a limit for the total number of training epochs to avoid runaway training. You should also perform integration tests after loading the trained model using sample inputs. Each trained model should be saved as a versioned artifact in your chosen artifact service, such as S3 or the MLflow Model Registry. Finally, seed random number generators to ensure deterministic training and log the seed. These practices help maintain consistency, reproducibility, and reliability across training runs.

Outputs

Trained model artifacts (pickle, ONNX, TensorFlow SavedModel)
Training logs and experiment metadata
Hyperparameter and dataset configuration snapshots

Validation, testing, and evaluation

Model evaluation starts with offline assessments using a holdout test dataset. In this stage, model performance is measured using task-appropriate measures. For classification, those measures may include accuracy, precision, recall, F1 score, ROC curve, and confusion matrix. For regression, common measures include RMSE, MAE, or R-squared. It is also necessary to evaluate domain-specific business metrics, such as conversion lift, cost of errors, or revenue impact, to ensure the deployed model provides business value in addition to statistical performance.

While offline assessment provides important deployment guidance, automated checks against predetermined thresholds or recorded baselines should be part of gated validation before promotion. These checks should validate fairness or bias issues and use unit tests to confirm that known inputs return expected outputs. If a required threshold is violated, the pipeline should fail and prevent the model from being promoted to production.

To maintain reliability, automate checks that compare metrics against defined thresholds or baselines. For example, the pipeline should fail if a model's accuracy falls below the previous version. The pipeline should also fail if a fairness metric for a protected group is violated. Include unit tests to confirm that the model produces correct predictions on known inputs. Only models that pass all validation checks should advance to deployment.

Outputs

Validation reports and evaluation metrics
Metric visualizations (confusion mat

AI Experimentation Best Practices: From Evaluation to Safe Production Rollouts

Tue, 02 Jun 2026 17:12:45 +0000

Introduction

Artificial intelligence tools, particularly large language models (LLMs), are not like traditional software. AI is probabilistic, so the same instructions and inputs can produce different results, especially when using non-zero temperature or other sampling methods, and those results can shift as your context changes. That unpredictability brings real risks because models can miss the mark, invent facts, or generate unfair or unsafe outputs. They can also incur unexpected costs and slow down under heavy loads, and they must constantly adapt to evolving policies and ethical guidelines.

AI experimentation means iteratively testing data, algorithms, prompts, models, and parameters to optimize model performance and validate hypotheses. You need a clear, repeatable way to try ideas, compare prompts and models, validate how your system finds and uses information, and do safety checks before changes reach real users. Experimentation is not just a nice-to-have; it is essential for shipping AI responsibly, optimizing resource efficiency, reducing costs, and accelerating innovation through rapid, evidence-based iteration cycles.

Throughout this guide, we distinguish evaluation from experimentation. Evaluation means offline benchmarking and scoring, including test sets, human or AI judges, and quality metrics. Experimentation means controlled production changes that affect real users through A/B tests, staged rollouts, or other release strategies. Evaluation tells you whether a variant clears a quality bar; experimentation tells you whether it beats the baseline in production, with statistical confidence and guardrails.

In this article, we cover the core ideas and practical steps for AI experimentation: how to plan a test, evaluate changes, run controlled trials with real users, choose metrics that actually matter to your product, and roll out changes safely. By the end, you will have a process that moves from initial concept to monitored, controlled production release.

AI Experimentation Best Practices

Best Practice	Description
Use experimentation to manage uncertainty	AI outputs can shift over time. Structured experimentation helps teams measure, compare, and validate changes before they reach users. LaunchDarkly AgentControl experiments and release options help turn unpredictability into a controlled process for improvement.
Build trust through evidence, not intuition	Without experimentation, teams rely on gut feeling. Controlled tests provide measurable evidence of what works. Use LaunchDarkly Experimentation, metrics, and AgentControl monitoring to make confident, data-driven decisions.
Detect and reduce hidden risks early	Experimentation surfaces hallucinations, bias, latency regressions, and safety failures before they affect broad audiences. Online evaluations and guarded rollouts help teams detect regressions and pause or roll back unsafe changes.
Enable continuous improvement	AI systems evolve as data, models, and contexts change. Config variations, config targeting, and progressive rollouts give teams a repeatable way to adapt while controlling exposure.
Design experiments with statistical power and variance in mind	Collect multiple observations per variant to account for nondeterminism. Use confidence intervals and statistical significance tests rather than single-run comparisons. Define a minimum detectable effect (MDE), guardrail metrics, and a decision rule before launch. LaunchDarkly experiments and metrics support this evidence-based workflow.
Support responsible and compliant AI	Experimentation frameworks help teams evaluate whether updates align with ethical standards, privacy requirements, and evolving policies. LaunchDarkly role-based access control, approvals, and audit logs help make responsible AI development a built-in process.
Keep track of cost and latency	Track per-session spend and speed, set budgets and max token limits, optimize prompts and context, use caching or streaming where appropriate, and monitor TTFT, p95/p99 latency, retries, and spend. AgentControl monitoring and autogenerated AI metrics help surface cost, latency, and token usage by variation.
Conduct controlled testing with real users	Run A/B tests, sticky cohorts, staged rollouts, or interleaving strategies. Measure satisfaction, task completion, latency, cost, and business impact. Use targeting rules, percentage rollouts, and guarded rollouts to control exposure and rollback thresholds.
Perform evaluation	Define metrics for truthfulness, user experience, reliability, safety, cost, and speed. Test in layers and expand only when stable. Evaluation tells you whether a system meets a bar, while experimentation determines which variant should be trusted in production. LaunchDarkly online evaluations, datasets, and judges support layered AI evaluation workflows.
Use retrieval evaluation for RAG	Evaluate model quality by measuring recall@k, precision@k, citation accuracy, unsupported claim rate, cost, and latency. After offline quality assessment, use live or shadow traffic for controlled experiments that optimize retrievers, chunking, ranking, or reranking. LaunchDarkly AgentControl experiments and monitoring help compare these changes safely.
Ensure proper governance and safety for AI experimentation	Pre-register your experiment plan, including hypothesis, primary metric, MDE, guardrails, and rollback rules. Version prompts, models, and configurations. LaunchDarkly config management, approvals, and audit logs help preserve compliance, safety, and auditability.

Note: Testing different chunking or embedding models usually requires building and validating separate vector indexes, and sometimes separate databases, because embeddings are tied to the index schema. Swapping these at inference time requires architectural planning, reindexing, and migration.

Why AI Needs Experimentation

Traditional software works like a calculator: same input, same output. AI is more like a conversational assistant that can be helpful and creative but sometimes surprising. Since AI is not fully predictable and small changes in wording can shift results, you cannot judge the quality of an AI feature from a single right answer.

AI features are pipelines with many moving parts: models that may update, prompts that steer behavior, tools and APIs that can fail, and knowledge sources that drift as content changes. All of these can affect accuracy, safety, speed, and cost. A one-time test will not catch issues that appear under real traffic.

That is why experimentation is essential. It gives teams a structured way to observe, measure, and improve AI behavior as conditions change. Through continuous testing, you can detect drift, uncover hidden risks, and build confidence that your system performs reliably and responsibly.

LaunchDarkly helps teams operationalize this workflow with AgentControl, configs, config variations, config targeting, monitoring, and online evaluations.

The Hierarchy of Levers: Where to Focus Your Optimization Efforts

In practice, AI experimentation levers should be optimized in order of impact and reversibility:

System message
Examples
Output format
Context
Retries and fallbacks
Models and parameters

This order matters because many high-impact changes can be made without retraining or rebuilding your system. With AgentControl config variations, teams can version and compare these changes while controlling exposure through targeting.

System Message Variations

The system message is one of the most powerful levers in shaping an AI model’s behavior. It defines the model’s role, tone, and boundaries, setting the personality and guardrails for how it responds.

Small changes here can dramatically affect safety and reliability. Tightening tone or adding an out-of-scope clause can prevent speculative or unsafe content. However, overly rigid instructions can make responses sound robotic or unhelpful.

Experiment with several system-message variations and test how they perform across normal, edge, and adversarial scenarios. The goal is not only to find one prompt that works, but to understand how tone and framing influence quality, cost, safety, and latency. Store and compare these variants with AgentControl config variations and monitor results with config performance monitoring.

Choosing the Right Number of Examples

Compare zero-shot, one-shot, and few-shot examples, typically 3-5 examples. Mix common cases and edge cases, include “do” and “don’t” examples, and show the exact output format. Short examples teach patterns, but they also add tokens and delay. Measure accuracy, format adherence, generalization, latency, and cost with autogenerated AI metrics.

Output Format

Choose between free text, structured templates, or native structured outputs. Structured outputs are easier to parse and validate but can constrain creativity or break on truncation. Always validate responses, handle partial outputs gracefully, and keep templates simple. During testing, a temporary explain field can help diagnose why one variation performs better than another.

Context Window Size

Your experiment should test the cost-benefit tradeoff between precise context and extended context. Increasing context often increases cost and latency without improving output quality. Use AgentControl monitoring to compare variation-level latency and token usage before promoting a longer-context variant.

Retries With Backoff

Use one or two attempts for temporary errors such as rate limits, timeouts, or server overload. Add exponential backoff and jitter. Log error rates, latency, and cost. Ensure idempotency, cap retries, enforce timeouts, and offer a polite fallback when limits are hit. For production rollout, pair retry changes with guarded rollouts so latency and error regressions can halt expansion.

Fallback Chain

Route to a backup model or provider in the event of failures or slowness. Keep prompts and formats aligned so the backup model understands the same prompt structure and returns responses in the same format. Preserve conversation state, verify required features on the fallback, and log reasons for routing. LaunchDarkly config targeting can help route different cohorts to different model or provider variations.

The Expansion Rule

Experimentation should scale based on evidence, not enthusiasm. Once your pilot shows strong performance, expand the rollout to broader audiences. Scale only when metrics justify it: success rates are high, failure rates are low, safety checks pass, and time or cost remains acceptable. Use percentage rollouts, progressive rollouts, or guarded rollouts to expand with controlled risk.

Models and Parameters

Models and parameters are the tuning panel for an AI system: the set of dials you use when you want more accuracy, fewer hallucinations, faster responses, or lower cost.

Start with the right model for the job. Use a more capable model for complex reasoning or planning and a smaller, faster model for routine tasks. Match the model’s strength to the complexity and stakes of the task rather than defaulting to the largest model. Lock down the exact model version when possible so results stay reproducible as the model evolves. Version pinning reduces variability, but it does not eliminate drift. Upstream model behavior and real-world inputs can still change, so production experiments and ongoing holdbacks remain necessary.

AgentControl lets teams manage model selection, prompt content, provider configuration, and generation parameters with configs, variations, and AI model configurations.

Temperature

Temperature controls how adventurous or conservative a model’s output is. It is the primary generation setting most users adjust.

Keep it low, around 0-0.3, for code, structured formats, or safety-critical tasks.
Use higher values, around 0.7-1.0, for creativity or brainstorming.
Stay in the middle for everyday conversations.

Other sampling parameters, such as top_p or top_k, also influence output diversity, but temperature usually has the largest and most predictable effect, so it is often the first parameter worth tuning.

Retrieval and Search

Do not rely only on keywords because meaning matters. Semantic search helps the model understand intent. Hybrid search, combining semantic and keyword search, often works best for short queries or exact names. Choose an embedding model that fits your language and domain, and keep its version fixed.

A graph database models relationships and traversals, such as “how is X connected to Y?” A vector database or vector-enabled datastore is optimized for similarity search over embeddings to support retrieval in RAG pipelines. When testing retrieval changes, use online evaluations and AgentControl experiments to compare quality, latency, and cost.

Chunking and Metadata

Split documents into natural sections with slight overlaps. Sliding windows help for long text. Add metadata to improve filtering and relevance. When experimenting, start with a baseline and change one variable at a time: temperature, chunk size, top_k, reranking, or search type. Evaluate offline using a labeled dataset from your domain, then use controlled rollout strategies such as percentage rollouts or guarded rollouts before broad exposure.

Tool and Function Management

Tools are the hands and eyes of your AI. They turn abstract intelligence into real-world action. However, giving an AI system too many tools at once can create reliability, safety, and cost problems. A focused, well-defined toolset keeps the system efficient and predictable.

When experimenting with tools, start small. Give the AI only the tools it truly needs, then expand based on evidence. Simulate tool behavior with mock or historical data before allowing live writes or sensitive operations. Monitor error rates, latency, and cost. Use circuit breakers, fallback paths, and kill switches to keep the system stable when a tool fails.

LaunchDarkly AgentControl tools, agents, feature flags, and release controls can help teams expose new tool behavior gradually and roll back unsafe changes quickly.

Cost and Latency

Managing cost and latency in AI systems is like tuning a race car: you want speed and performance, but you cannot afford to burn all your fuel in one lap. The trick is knowing where your money and time actually go: input tokens, output tokens, model rates, tool usage, retries, retrieval, and post-processing.

Experiment design also affects cost. Multi-armed bandit approaches can reduce spend by shifting traffic away from losing variants early, while long, fixed-horizon A/B tests can waste budget after a clear loser emerges. Track cost per successful answer rather than cost per call so you know which variants are efficient and useful.

Several habits help:

Match the model to the job: Use smaller models for routine tasks and larger models for complex reasoning.
Set clear budgets: Cap tokens, cost, and retries per session.
Cache and reuse: Avoid paying twice for the same retrieval or generated output.
Retry wisely: Validate inputs early and use exponential backoff to avoid waste.
Measure what matters: Track cost per successful answer, not just cost per request.
Watch latency signals: Monitor time to first token, p95/p99 latency, and error rates.

LaunchDarkly Monitoring and autogenerated AgentControl metrics help teams compare token usage, duration, and variation-level performance.

Experimentation Before User Exposure

Before any major AI update reaches real users, it deserves a proper dress rehearsal. Catching issues early prevents bad experiences, unnecessary costs, and reputational damage.

Start by building a test set that mirrors real-world scenarios: genuine examples, synthetic edge cases, and adversarial prompts. If you are working with RAG, make sure

AI Governance in Practice: FastAPI on EKS with Model Cards, Audit Logging, and Helm

Wed, 13 May 2026 13:53:19 +0000

AI governance is increasingly a business requirement, not an afterthought. Whether it's the EU AI Act, NIST AI RMF, or an internal risk committee, the question is the same: can you prove your model is behaving as intended, on every request, with a documented audit trail?

This post walks through an AI governance platform I built on AWS EKS: a FastAPI service that runs churn inference, records every prediction to an audit log, exposes a machine-readable model card, and packages everything into a Helm chart with horizontal pod autoscaling.

Source code: github.com/tsekatm/eks-ai-governance

Architecture

Client
  └── POST /predict → FastAPI (EKS pod)
                           │
                           ├── LogisticRegression inference
                           ├── Audit log entry (request_id, features, result)
                           └── Response: churn_probability, prediction, request_id

Kubernetes (EKS)
  ├── Helm chart  →  Deployment + Service + HPA (2–10 replicas)
  └── Terraform   →  VPC, EKS cluster, node groups, IAM OIDC

Governance Layer
  ├── GET /governance/model-card   → version, metrics, fairness, EU AI Act tier
  └── GET /governance/audit-log   → full prediction audit trail

The app is self-contained: no external database needed to demo it. Swap the in-memory audit log for DynamoDB and the model for a SageMaker endpoint and this is production-ready.

Step 1: The FastAPI Application

Four responsibilities, four endpoint groups.

Inference (`POST /predict`)

class PredictRequest(BaseModel):
    customer_id: str
    tenure_months: float
    monthly_charges: float
    total_charges: float
    num_complaints: int

@app.post("/predict", response_model=PredictResponse)
def predict(request: PredictRequest) -> PredictResponse:
    features = np.array([[
        request.tenure_months,
        request.monthly_charges,
        request.total_charges,
        request.num_complaints,
    ]])
    scaled = _scaler.transform(features)
    churn_prob = float(_model.predict_proba(scaled)[0][1])
    prediction = "churn" if churn_prob >= 0.5 else "retain"
    request_id = str(uuid.uuid4())

    audit_log.append({
        "request_id": request_id,
        "customer_id": request.customer_id,
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "features": request.model_dump(exclude={"customer_id"}),
        "prediction": prediction,
        "churn_probability": churn_prob,
        "model_version": MODEL_VERSION,
    })

    return PredictResponse(
        customer_id=request.customer_id,
        churn_probability=round(churn_prob, 4),
        prediction=prediction,
        model_version=MODEL_VERSION,
        request_id=request_id,
        timestamp=datetime.now(timezone.utc).isoformat(),
    )

Every prediction writes to the audit log before returning. The request_id ties the response to the log entry — useful for compliance queries: "show me exactly what the model returned for customer X on date Y."

Model Card (`GET /governance/model-card`)

A machine-readable model card makes governance reviewable by automated tools, not just humans with PDFs.

MODEL_CARD = {
    "model_id": "telecom-churn-v1",
    "version": "1.0.0",
    "type": "LogisticRegression",
    "training_date": "2026-04-29",
    "features": ["tenure_months", "monthly_charges", "total_charges", "num_complaints"],
    "metrics": {
        "accuracy": 0.89,
        "roc_auc": 0.92,
        "precision": 0.85,
        "recall": 0.81,
    },
    "fairness": {
        "demographic_parity_difference": 0.03,
        "equalized_odds_difference": 0.02,
        "evaluation_date": "2026-04-29",
    },
    "governance_tier": "Medium Risk",
    "eu_ai_act_classification": "Limited Risk",
    "approved_by": "AI Governance Board",
    "next_review": "2026-10-29",
}

The governance_tier and eu_ai_act_classification fields are what a risk committee actually needs. A CI gate could fail a deployment if these fields are missing or if next_review is in the past.

In a telecoms context, POPIA (South Africa) and RICA compliance add additional constraints on what customer data can flow through inference pipelines, making PII-aware audit logging essential. The audit log here records features and predictions but not raw PII — in production, customer_id would be a pseudonymised identifier resolved only by authorised downstream systems.

The fairness metrics (demographic_parity_difference, equalized_odds_difference) are hardcoded for this demo. In production, they are computed during the evaluation step of the SageMaker Pipeline (see Part 2 of this series) against protected attributes and updated automatically per training run before the model card is published.

Audit Log (`GET /governance/audit-log`)

@app.get("/governance/audit-log")
def get_audit_log() -> dict:
    return {"total": len(audit_log), "entries": list(audit_log)}

Simple. In production this would be backed by DynamoDB with a device_id + timestamp key schema — the same pattern used in the IoT pipeline project.

Health probes

@app.get("/healthz")
def healthz():
    return {"status": "ok", "model_version": MODEL_VERSION}

@app.get("/ready")
def ready():
    return {"status": "ready", "model_id": MODEL_ID}

Two separate probes because Kubernetes needs them for different purposes: /healthz is the liveness check (restart if down), /ready is the readiness check (remove from service if not ready to serve traffic). The readiness probe path is configurable in Helm values.

Step 2: The Model

The model is a LogisticRegression trained on 2,000 synthetic telecom accounts at module load time:

_rng = np.random.default_rng(42)
_X_train = np.column_stack([
    _rng.uniform(1, 72, 2000),     # tenure_months
    _rng.uniform(20, 150, 2000),   # monthly_charges
    _rng.uniform(20, 10000, 2000), # total_charges
    _rng.integers(0, 8, 2000),     # num_complaints
])
# Rule: ≥3 complaints, or new customer + high charge → churn
_y_train = (
    (_X_train[:, 3] >= 3) |
    ((_X_train[:, 0] < 6) & (_X_train[:, 1] > 80))
).astype(int)

_scaler = StandardScaler()
_model = LogisticRegression(max_iter=200, random_state=42)
_model.fit(_scaler.fit_transform(_X_train), _y_train)

Replacing this with a SageMaker endpoint call is a one-function swap — the governance layer (audit logging, model card) stays identical.

Step 3: Test Suite — 23/23

All tests written before implementation (TDD).

tests/test_app.py::TestHealthEndpoints        4 passed
  - /healthz returns 200 with status "ok"
  - /ready returns 200 with status "ready"

tests/test_app.py::TestPredictEndpoint        9 passed
  - 200 on valid payload
  - customer_id, churn_probability (0–1), prediction, model_version, request_id present
  - 422 on missing field
  - high-risk profile (1 month tenure, 5 complaints) → churn
  - low-risk profile (60 months, 0 complaints) → retain

tests/test_app.py::TestAuditLog               5 passed
  - empty initially
  - records entry after prediction
  - entry contains customer_id and prediction
  - total count increments correctly

tests/test_app.py::TestModelCard              5 passed
  - 200 response
  - version, metrics, fairness, governance_tier present

The fixture autouse=True on clear_audit_log ensures each test class starts with a clean in-memory log — no test bleeds state into the next.

Step 4: Docker — Multi-Stage Build

FROM python:3.12-slim AS builder
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends gcc libpq-dev
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt

FROM python:3.12-slim AS production
RUN groupadd -r appuser && useradd -r -g appuser -d /app -s /sbin/nologin appuser
WORKDIR /app
COPY --from=builder /install /usr/local
COPY . .
RUN chown -R appuser:appuser /app
USER appuser
EXPOSE 8080
HEALTHCHECK CMD curl -f http://localhost:8080/healthz || exit 1
CMD ["python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"]

Two stages: the builder installs build tools and compiles packages; production copies only the compiled artifacts. Result: 146 MB image, non-root user, health check baked in.

docker build --platform linux/amd64 -t ai-governance-platform:local .
# Successfully built d1a906f056b5 — 146MB

Step 5: Helm Chart with HPA

The chart packages deployment + service + HPA into a single parameterised artifact.

HPA

# helm/templates/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: {{ .Values.autoscaling.minReplicas }}   # 2
  maxReplicas: {{ .Values.autoscaling.maxReplicas }}   # 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}   # 70
    - type: Resource
      resource:
        name: memory
        target:
          averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }} # 80

The HPA scales between 2 and 10 pods on CPU (70%) and memory (80%) utilisation. Both thresholds are overridable per environment via values.yaml.

`_helpers.tpl`

Helm charts need a _helpers.tpl file to define shared template functions like fullname and labels. Without it, the templates can't render:

{{- define "ai-platform.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}

helm lint ./helm/
# ==> Linting ./helm/
# 1 chart(s) linted, 0 chart(s) failed

Deploy:

helm install ai-platform ./helm -f helm/values.yaml
kubectl get pods    # 2–10 replicas depending on load
kubectl get hpa     # watch scaling events

Step 6: Infrastructure (Terraform)

The EKS cluster is provisioned with Terraform — nothing clicked in the console.

module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "~> 20.0"
  cluster_name    = "ai-governance-cluster"
  cluster_version = "1.30"
  vpc_id          = module.vpc.vpc_id
  subnet_ids      = module.vpc.private_subnets

  eks_managed_node_groups = {
    general = {
      instance_types = ["m5.large"]
      min_size       = 2
      max_size       = 6
      desired_size   = 3
    }
  }

  enable_irsa = true
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"
  name    = "ai-governance-vpc"
  cidr    = "10.0.0.0/16"

  azs             = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway = true
}

enable_irsa = true provisions the OIDC provider so the ai-platform-sa service account (created by Helm) can assume an IAM role for DynamoDB and CloudWatch access — no static AWS credentials in the pod.

What I'd Add Next

SageMaker endpoint — replace the in-memory model with boto3.client("sagemaker-runtime").invoke_endpoint(), zero governance layer changes needed
DynamoDB audit store — replace the in-memory list with a DynamoDB table (model_id + timestamp key), enables cross-pod audit queries
Bedrock Guardrails — content filtering and PII redaction on inference inputs before they hit the model; a pattern I've already implemented in production for prompt injection prevention
CI gate on model card — fail deployment if next_review is expired or governance_tier is missing
Prometheus metrics — prediction_count, churn_rate, p99_latency scraped by CloudWatch Container Insights

Tebogo Tseka — Cloud Solutions Architect & ML Engineer
GitHub: @tsekatm | Blog: tebogosacloud.blog

1minMLOps #2 :Versioning your data with DVC

Wed, 13 May 2026 00:40:36 +0000

In the last article we talked about why ML is harder than regular software: code, data and environment all move at the same time. Today we're tackling the second one data with a tool called DVC (Data Version Control).

Why not just use Git?

Git is amazing for code, but it was designed for small text files. The moment you commit a 2 GB CSV or a folder of 50,000 images, things get unpleasant fast: the repo balloons, git clone becomes a coffee break, and GitHub starts politely asking you to leave.

DVC solves this by being "Git for data": it stores tiny pointer files in your repo and pushes the actual heavy data to a separate storage backend (S3, GCS, an SSH server, even a local folder). You get versioning, branching and reproducibility, without bloating Git.

Step 1: Install DVC

pip install dvc

If you want S3 support, install the extra:

pip install "dvc[s3]"

Other backends like gs, azure, ssh work the same way — just swap the extra.

Step 2: Initialize DVC in your project

Let's start a tiny project:

mkdir mlops-demo && cd mlops-demo
git init
dvc init
git commit -m "Initialize DVC"

dvc init creates a .dvc/ folder (a bit like .git/) and a .dvcignore file. From now on, DVC and Git work side by side.

Step 3: Track your first dataset

Let's grab a small dataset to play with. We'll use the classic Iris CSV:

mkdir data
curl -L https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv -o data/iris.csv

Note (PowerShell users): in PowerShell, curl is an alias for Invoke-WebRequest, which doesn't accept the -L flag and will error with A parameter cannot be found that matches parameter name 'L'. Use one of these instead:
# Option 1: call the real curl binary (ships with Windows 10+)
curl.exe -L https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv -o data/iris.csv

# Option 2: native PowerShell
Invoke-WebRequest -Uri https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv -OutFile data/iris.csv
curl.exe follows redirects by default, so -L is optional there.

Now tell DVC to track it:

dvc add data/iris.csv

DVC will:

Move data/iris.csv into its cache (.dvc/cache/)
Create a small pointer file data/iris.csv.dvc
Add data/iris.csv to .gitignore automatically

Commit the pointer, not the data, to Git:

git add data/iris.csv.dvc data/.gitignore
git commit -m "Track iris dataset with DVC"

If you peek inside data/iris.csv.dvc, you'll see something like:

outs:
- md5: 1f8e3c...
  size: 3858
  hash: md5
  path: iris.csv

That hash is the version of your data. Change one byte in the CSV, and the hash changes.

Step 4: Set up a remote storage

Right now, the data only lives on your machine. Let's push it somewhere others (or future-you on another laptop) can pull it from.

For a quick local test, you can use a folder as a "remote":

mkdir -p /tmp/dvc-storage
dvc remote add -d localremote /tmp/dvc-storage
git add .dvc/config
git commit -m "Configure DVC remote"

For real projects, swap that with S3 or similar:

dvc remote add -d s3remote s3://my-bucket/dvc-storage

Then push the data:

dvc push

Step 5: The reproducibility test

This is the moment that makes DVC click. Let's pretend you're a teammate cloning the repo for the first time:

cd /tmp
git clone /path/to/mlops-demo fresh-clone
cd fresh-clone
ls data/
# Only iris.csv.dvc — the actual CSV is missing!

dvc pull
ls data/
# iris.csv is back, byte-for-byte identical

You just versioned a dataset alongside your code, without committing it to Git.

Step 6: Updating the dataset

Real data changes. Let's simulate that:

echo "6.0,3.0,4.5,1.5,versicolor" >> data/iris.csv
dvc add data/iris.csv
git add data/iris.csv.dvc
git commit -m "Add new sample to iris dataset"
dvc push

The pointer file's hash updated. If you ever need the old version of the data, just git checkout an older commit and run dvc pull , DVC fetches the dataset that matched that commit. Time travel for data.

Why this matters

With this in place, you can finally answer the question "which data produced that model?" with a Git commit hash. That's a huge upgrade.

In the next article, we'll add the second piece of the puzzle: experiment tracking with MLflow, so we never again lose track of which hyperparameters and which data produced which metric.

Stay tuned and have fun!

If you enjoyed this article, you can support my work here:

1minMLOps #1 : What is MLOps and why should you care?

Wed, 13 May 2026 00:40:09 +0000

If you've ever trained a beautiful model in a Jupyter notebook, watched the metrics shine, and then realized you have no idea how to actually put it in front of users, congratulations: you've just discovered why MLOps exists.

In this series, we are going to walk together from a notebook to a fully deployed, monitored and self-retraining ML system, one tiny step at a time. But before we write any code, let's get the foundations straight

So, what is MLOps?

MLOps (short for Machine Learning Operations) is the set of practices, tools and culture that lets you ship machine learning models to production reliably and repeatedly. Think of it as DevOps' younger sibling: same spirit (automation, reproducibility, monitoring), but adapted to the weirdness of ML, where your code is not the only thing that changes, your data changes, your model changes, and the world your model lives in changes too

A useful way to picture it is the ML lifecycle:

Data collection & versioning — where does the data come from, and which version did we train on?
Experimentation — which features, which model, which hyperparameters?
Training & evaluation — does it actually work, and is it better than what we had?
Packaging — wrap the model in something deployable
Deployment — serve predictions to real users (batch or real-time)
Monitoring — is it still working? Did the data drift?
Retraining — close the loop and start again

Traditional software has steps 4–6. ML has all seven, and steps 1–3 keep coming back to haunt you

Why "it works on my machine" is worse in ML

In classical software, if your code runs locally, it has a decent chance of running in production. In ML, that's a trap, because the model's behavior depends on three moving things, not one:

Code: the training script, the preprocessing, the inference logic
Data: the exact dataset (and its version) you trained on
Environment: Python version, library versions, CUDA versions, OS

Change any of these three and your "great model from Tuesday" becomes "mysterious garbage on Friday" This is why ML teams need stricter versioning, tracking and packaging discipline than most web teams.

What problems does MLOps actually solve?

Concrete pains you'll feel without MLOps, and that we'll fix in this series:

"Which dataset gave us that 0.94 F1 score? Nobody remembers."
"The model works locally but crashes in the Docker container."
"We retrained the model and accuracy dropped, but we can't roll back."
"Production is silently degrading and we noticed two weeks later."
"Every deploy is a hand-crafted artisanal disaster."

Each of these has a tool and a workflow that solves it, and we are going to meet them(almost) one by one

The MLOps stack we'll build

Here's a sneak peek of the tools we'll touch in the next articles:

DVC for data versioning
MLflow for experiment tracking and the model registry
FastAPI for serving
Docker for packaging (we'll lean a bit on Clelia's 1minDocker series here)
GitHub Actions for CI/CD
Evidently for monitoring data and model drift (we can use prometheus and grafana too)
A cloud provider (we'll pick one later) for actually deploying it all

Don't worry if some of these names sound intimidating, we'll introduce them gently, one per article, and always with a working example.

What you need to follow along

Nothing fancy:

Python 3.10+
git installed
A GitHub account
Docker installed (highly recommend to follow this series https://dev.to/astrabert/1mindocker-1-what-is-docker-3baa)
A laptop and ~1 minute per article

In the next article, we'll get our hands dirty: we'll take a small dataset, version it with DVC, and finally answer the question "which data did we train on?" without crying

Stay tuned and have fun!

Yevideo - AI Video and Image Platform

Fri, 08 May 2026 04:58:00 +0000

Yevideo is an AI creation platform for content marketing and creative teams, offering text-to-video, image-to-video, video-to-video, AI video editing, text-to-image, and image-to-image capabilities. Users can create, edit, and reuse assets in a unified workspace to shorten production cycles and improve publishing efficiency. The platform is suitable for social media operations, eCommerce assets, brand advertising, and creative testing, helping teams produce high-quality visual content quickly. For businesses that require continuous content updates, Yevideo lowers production barriers while increasing output and iteration speed.

Stop Treating Your Data Pipeline Like a Script - Treat It Like a Product

Mon, 27 Apr 2026 19:04:09 +0000

I learned this the hard way after watching a simple ETL job torch our weekend.

When I started in data engineering, I thought my job was writing scripts that moved data from A to B. Clean, logical, done. I was wrong.

The difference between a pipeline that works and one that survives? Three things nobody told me:

Observability first, logic second - If you can't see what's happening inside your pipeline, you're flying blind. Dashboards aren't optional; they're infrastructure.

Data contracts over hope - Assume your upstream source will silently betray you. Schema changes, null explosions, timestamp format switches at 2am. Code defensively or suffer.

Idempotency is non-negotiable - Rerunning yesterday's job shouldn't duplicate records or corrupt state. Build for reruns, not just first runs.

The mindset shift: Your pipeline isn't finished when it runs. It's finished when it runs reliably while you're sleeping.

What's one lesson you learned after your first production failure? Drop it below.

MCP Explained: The Protocol That’s Quietly Rewiring How AI Works

Sat, 25 Apr 2026 06:36:16 +0000

MCP is not a buzzword. It might be the most underrated infrastructure shift happening in AI right now.

Most people are still debating which AI model is smartest. Meanwhile, something much quieter — and arguably more consequential — has been taking shape in the background.

It’s called MCP. Model Context Protocol. And if you haven’t heard of it yet, you will.

Before you tune out because it sounds like another acronym someone invented to feel important, give it a moment. What MCP is actually about isn’t really technical. It’s about something more fundamental: how AI connects to the world around it, and what changes when it finally can.

So, What Even Is MCP?

Here’s the honest answer: MCP is a standard. A common language that lets AI models talk to external tools, data sources, and services without each connection needing to be custom-built from scratch.

Think of it like USB. Before USB, every device had its own proprietary port. Your keyboard plug didn’t fit your printer, your printer cable didn’t fit anything else, and setting up a new peripheral was a minor ordeal. USB came along and said: one standard, plug it in, it works. MCP is trying to do that for AI.

Right now, if you want an AI assistant to read from your Google Drive, check your calendar, query a database, and send a Slack message — each of those connections needs to be individually plumbed in. It’s fragile, bespoke, and expensive to maintain. MCP proposes a unified interface: one standard that any AI model can use to connect to any tool that supports it.

Anthropic introduced the protocol in late 2024. Since then it’s picked up real momentum — Cursor adopted it, Zed added support, and a fast-growing library of community-built MCP servers now covers everything from GitHub to Notion to Stripe. It’s still early. But the foundations are being laid quickly.

Why Is Everyone Suddenly Talking About It?

The timing makes sense if you zoom out.

We’ve spent two years marveling at what AI models can do in isolation — write code, draft emails, explain complex ideas. But isolated AI is limited AI. A model that can’t access real-time data, take actions in the world, or read your actual files is useful in a narrow way. Impressive in a demo. Less impressive on a Tuesday when you actually need something done.

The real unlock for AI isn’t making models smarter — it’s giving them reach. The ability to actually do things, not just say things.

That’s what’s driving the excitement around MCP. It’s infrastructure for agentic AI — AI that acts, not just responds. As more people build agents that book meetings, write and deploy code, or manage workflows end-to-end, the question of how those agents connect to everything else becomes urgent. MCP is one serious answer to that question.

There’s also a network effect at play. The more tools support MCP, the more useful every MCP-compatible AI becomes. It’s not just a protocol — it’s the beginning of an ecosystem.

The Good, The Complicated, and The Honest Caveats

MCP has real strengths. It’s open — Anthropic released the spec publicly, no licensing fees. That matters enormously for adoption. It reduces integration complexity dramatically: instead of building ten custom connectors, a developer builds one MCP server and becomes compatible with any AI that supports the protocol. It’s also model-agnostic, meaning this isn’t a bet on Claude specifically. Any model from any company could adopt the same standard.

But it’s not a clean story yet.

The tooling ecosystem is thin and documentation is still catching up. Security is a genuine concern — giving AI agents broad access to external systems raises real questions about what they can do, what they can see, and what happens when something goes wrong. These aren’t hypothetical problems. They’re the kind that show up when you move from prototype to production.

There’s also a fragmentation risk. Other protocols could emerge. Big players could decide they’d rather build walled gardens than support an open standard. The history of tech is littered with promising open standards that never quite made it. MCP could still go that way.

None of this means it won’t succeed. It just means the outcome isn’t written yet.

What Does This Actually Change for the People Building Things?

If you’re building products with AI, MCP is worth paying close attention to right now — not because it’s mature, but because foundational infrastructure rewards early familiarity. Getting comfortable with it now is how you avoid painful rewrites later.

Practically, it changes the design question. Instead of asking what a model can do, you start asking what it can access. That’s more interesting. An agent that reads your CRM, checks inventory, updates a ticket, and sends a confirmation email isn’t magic. It’s plumbing. Good plumbing changes what’s possible.

For people leading teams or making product decisions without writing code: MCP is part of why AI assistants are about to get significantly better at multi-step tasks. The jump from AI that answers questions to AI that completes workflows is coming faster than most roadmaps account for. Worth asking now whether your team is actually ready for what that means.

Where Is This Going?

If MCP achieves broad adoption — still an if — it changes the calculus on AI agents fundamentally. Building agents that work across complex, multi-system workflows becomes significantly easier. The barriers drop, more people build, more tools become MCP-compatible. Flywheels do what flywheels do.

There’s also a downstream effect most people aren’t thinking about yet: broad MCP adoption could shift where the value in AI actually sits. If models become increasingly interchangeable and tools become plug-and-play, competitive advantage moves toward who has the best data, the most useful integrations, and the sharpest sense of what workflows to automate. The protocol, in other words, could commoditize the model layer and elevate everything around it.

That’s a significant restructuring of the AI market. Most people are still focused on benchmark scores.

There’s one more thing worth flagging — and it’s underreported. MCP could become foundational infrastructure for multi-agent systems. Right now, most AI deployments are one model, one task. The emerging architecture is multiple specialized agents coordinating together: one researches, one writes, one reviews, one executes. For that to work at scale, those agents need a shared way to access tools and context. MCP is a credible candidate for that connective tissue. If that happens, what looks like a developer convenience today becomes something closer to the nervous system of enterprise AI.

” The competitive advantage may shift toward who has the best data and
the sharpest sense of what to automate — not who has the smartest
model.”

Final Take

MCP won’t show up on your radar the way a new model release does. No dramatic demo, no benchmark, no announcement video. It’s infrastructure. Infrastructure is boring right up until it isn’t — and then suddenly it’s the thing everything else is built on.

The models are getting smarter. But the bigger shift right now might be simpler than that: they’re getting connected. And connected, as it turns out, is what makes intelligence actually useful.

Thanks for reading. If this sparked a thought, leave a reply — I read every one.

How a Revolutionary EdTech Platform Is Helping Romania Keep Pace with the AI Revolution

Tue, 14 Apr 2026 21:29:26 +0000

The global AI race is accelerating — and countries that fail to upskill their workforce risk falling behind permanently. While Western Europe and the US dominate AI adoption, Romania is writing its own chapter in the story, thanks to a bold EdTech initiative that's making AI education accessible, practical, and affordable.

The Problem: A Growing AI Skills Gap in Eastern Europe
According to the European Commission's Digital Economy and Society Index (DESI), Romania has consistently ranked among the lowest EU member states in digital skills. While the country boasts a thriving IT outsourcing sector and world-class software engineers, AI literacy among professionals outside of tech remains critically low.

The challenge isn't talent — Romania produces thousands of STEM graduates every year. The challenge is access to structured, practical AI education in a format that professionals can actually use. Most international AI courses are expensive, English-only, and overly theoretical. For a Romanian marketing manager, HR director, or entrepreneur looking to integrate AI into their workflow, the barrier to entry has been steep.

The Solution: Cursuri-AI.ro — Romania's First Premium AI Education Platform
Enter Cursuri-AI.ro — a platform built from the ground up to solve this exact problem.

Launched by GALIAN SOFTWARE DEVELOPMENT SRL, a Cluj-Napoca-based software company, Cursuri-AI.ro is Romania's first dedicated AI education platform offering a full library of practical courses in Romanian, accessible through a single subscription.

What Makes It Different? ✨
Here's what sets this platform apart from generic online course marketplaces:

28 Practical Courses and Growing
The platform currently offers 28 structured courses covering everything from AI fundamentals and prompt engineering to advanced applications in marketing, business automation, content creation, and data analysis. Every course is designed for real-world application, not academic theory.

Integrated Virtual AI Tutor
This is the game-changer. Cursuri-AI.ro features a built-in AI professor that guides each student step by step, answers questions in real time, and personalizes the learning experience. It's like having a private AI mentor available 24/7 — something no traditional online course platform offers at this scale.

100% in Romanian
Language shouldn't be a barrier to learning. All courses are delivered in Romanian with proper terminology, making complex AI concepts accessible to professionals who aren't comfortable learning in English.

One Subscription, Full Access
No hidden fees, no per-course pricing. A single subscription unlocks the entire library — past, present, and future courses. This model removes the friction of "which course should I buy?" and encourages continuous learning.

Why This Matters for Romania's Future
Romania is at a crossroads. The country has several structural advantages in the AI era:

️ Strong IT infrastructure — Romania ranks among Europe's top countries for internet speed and has a mature tech outsourcing industry
STEM talent pipeline — top universities in Cluj-Napoca, Bucharest, Iași, and Timișoara produce thousands of technical graduates annually
Cost competitiveness — Romanian companies can adopt AI at a fraction of the cost of Western European counterparts
EU membership — access to EU funding programs, Digital Europe Programme, and Horizon Europe grants for AI adoption
But these advantages mean nothing without a workforce that understands how to use AI tools effectively. And that's exactly the gap that platforms like Cursuri-AI.ro are filling.

The Numbers Tell the Story
The global AI education market is projected to reach $20 billion by 2027 (Source: Global Market Insights)
Romania's IT sector contributes approximately 6.2% of GDP, one of the highest rates in the EU
Over 70% of Romanian companies plan to integrate AI tools by 2027, according to a Deloitte CE survey
Yet fewer than 15% of Romanian professionals report having formal AI training
The demand is massive. The supply of quality, localized AI education is scarce. Cursuri-AI.ro is positioning itself as the bridge.

A Model for Eastern European EdTech
What's happening in Romania could serve as a blueprint for the entire CEE region. Countries like Poland, Czech Republic, Hungary, and Bulgaria face similar challenges: strong tech sectors, growing AI adoption, but a shortage of accessible, localized AI education.

By proving that a premium, subscription-based AI education platform can work in Romania — a market often considered too small or too price-sensitive for premium digital products — Cursuri-AI.ro is demonstrating that quality EdTech doesn't have to come from Silicon Valley.

Final Thoughts
The AI revolution won't wait for countries that aren't ready. But readiness isn't about having the biggest budgets or the most advanced research labs — it's about empowering everyday professionals with the skills they need to work alongside AI.

Romania has the talent, the infrastructure, and now — with platforms like Cursuri-AI.ro — the educational ecosystem to compete. The question isn't whether Romanian professionals can learn AI. It's whether they'll start today. ⏳

Interested in exploring Romania's premier AI education platform? Visit cursuri-ai.ro to browse the full course library and meet your virtual AI tutor.

How Docker Sandboxes making local AI Agents safer

Wed, 08 Apr 2026 05:12:38 +0000

Let an AI agent work on your machine and things can go wrong fast:

Accessing files it shouldn’t
Leaking secrets
Running destructive commands
Modifying things you never intended

So teams do the obvious thing: Add guardrails

But Guardrails inside the agent slow it down.

What agents actually need is:

A clear boundary before execution
A safe environment to operate freely inside

Think of it like this:

Don’t control every move.

Control the playground.

This is where Docker Sandboxes propose to change the game

Docker Sandboxes give agents the freedom to operate…

without giving them access to everything.

Instead of restricting the agent…

You isolate it.

Runs in its own microVM
No access to your system unless you allow it
No shared state, no accidental leaks
Spins up in seconds, disappears after the task

Works with everything you already use

Claude Code, Copilot CLI, Codex, Gemini, OpenClaw…

No new workflow. Just a safer environment.

Give it a try and let me know what you think about Docker Sandboxes.

The Butterfly Effect in Networked Agents

Tue, 31 Mar 2026 16:06:33 +0000

Imagine your bank's customer support is entirely run by AI agents. A triage agent classifies your issue and routes you to specialized agents — billing, fraud, technical support — with an escalation agent handling complex disputes.

Your bank upgrades the triage agent. The new model is smarter — edge cases that used to get misrouted to billing now correctly go to the fraud team. Every test passes. Accuracy improves. The upgrade is a success.

Within a week, the fraud agent is drowning. Its volume doubles, and half the new queries are ambiguous cases that it was never calibrated for. False positives spike. Legitimate customers get their accounts frozen. The downstream escalation agent, designed for rare, complex disputes, is flooded with angry customers who can't access their money. Resolution times triple.

Every agent is doing exactly what it was designed to do. No agent is broken. The system is broken.

Welcome to the butterfly effect in networked agents — a small improvement in one agent silently reshaping the operating conditions of every agent downstream. Not by sending wrong data, but by sending a different distribution of correct data.

Why This Is Different From Traditional Software

In traditional software, when you change the behavior of an upstream service, you think about the downstream impact. You version your API. You communicate breaking changes. You run integration tests.

But this isn't a breaking change. The triage agent's API didn't change. Its output schema is identical. The downstream agents receive the same data types in the same format. What changed is the statistical profile of the requests flowing through the system — and we have no tooling to version, communicate, or test for that.

Traditional monitoring doesn't help either. The fraud agent's error rate went up, so you investigate. Its model is fine. Its prompts are fine. You A/B test new prompts, retrain on fresh data, and tweak thresholds. But it still won't match its previous performance, because it was dealing with simpler, pre-filtered cases before. The problem isn't the fraud agent. The inputs changed, not the agent.

This isn't a new problem, strictly speaking. Data drift has been a known challenge in machine learning for years — models degrade when their input distribution shifts away from their training data, whether due to upstream changes or seasonal shifts in user behavior. But historically, it stayed low-priority. ML-based systems were mostly self-contained. A recommendation engine, a fraud detector, a search ranker — each operated within a single cluster in companies, fed by data pipelines that teams controlled. Building and deploying ML systems required specialists who could train models and maintain infrastructure. ML systems were rarely chained together, so distributional shifts were rarely caused by upstream ML system updates.

Agents change the equation entirely. The barrier to building and deploying an agent is a fraction of what it takes to build traditional ML software. You don't need a team of ML engineers to train and deploy a model — you need a prompt and an API key. Every company can now embed agents in every component of its software. And once agents are everywhere, networking them becomes inevitable. When agents from different companies are chained together, the input distribution of every downstream agent is shaped by the behavior of upstream agents it doesn't control, can't monitor, and may not even know about. Data drift goes from an internal ops concern to an inter-organizational reliability problem — and the tooling hasn't caught up.

This is the fundamental asymmetry: their improvement is your regression. The triage agent got better at its job, and that made every downstream agent worse at theirs. Not because anything broke, but because the downstream agents were implicitly calibrated to the old distribution, and nobody knew that was a dependency.

Now, Imagine This Across Companies

The call center example happens inside one organization. That means one team can, in theory, detect the problem, trace the cause, and recalibrate the chain — even if it takes weeks.

Now imagine the agents are owned by different companies.

The triage agent is a third-party service your bank subscribes to. The fraud detection agent is from another vendor. The escalation routing goes through a customer experience platform built by a fourth company. Each company maintains its own agent, model, and deployment schedule.

The triage vendor upgrades their model. Your fraud vendor's agent starts underperforming. Your fraud vendor investigates, finds nothing wrong, and blames your data. You escalate to the CX platform vendor, which sees degraded metrics but has no access to the fraud or triage models. The triage vendor, meanwhile, is celebrating improved accuracy numbers and has no idea their upgrade destabilized two downstream systems they've never heard of.

Nobody can see the full chain. Nobody has the access, the context, or the incentive to diagnose a cross-boundary distributional shift. The problem festers until someone manually connects the dots — or until it shows up in quarterly customer churn numbers, triggering a post-mortem that takes months.

This is where the butterfly effect becomes truly dangerous. Inside one company, it's an operational headache. Across companies, it's an invisible failure with no owner.

What the Industry Needs

We spent decades building reliability infrastructure for traditional software composition — type systems, semantic versioning, contract testing, and dependency management. For networked agents, we have nothing equivalent. Here's what needs to exist.

Distribution-aware contracts. Agent-to-agent interfaces need to specify not just data types but expected input distributions. "This agent expects 80-90% of inputs to be likely fraud cases" is a distributional contract. If the upstream agent's behavior shifts that distribution outside the agreed range, it should be treated as a breaking change — even if the schema is identical.
Dependency-aware deployment. When you upgrade one agent, the deployment pipeline should automatically identify which downstream agents are affected and trigger re-evaluation. This is the agent equivalent of "which services depend on this one?" — except the dependency isn't in the API contract, it's in the statistical properties of the data flowing between them.
Continuous distributional monitoring. Every agent should track the statistical profile of its inputs over time — volume, composition, feature distributions. When the profile shifts beyond a threshold, alert. This is data drift monitoring applied to agent communication, and it should be as standard as uptime monitoring.
Automated recalibration pipelines. When a distributional shift is detected, downstream agents should have automated pipelines to re-run evaluations, adjust thresholds, update few-shot examples, or trigger fine-tuning. Manual recalibration doesn't scale — especially across organizational boundaries.

What You Can Do Today

The industry-level solutions don't exist yet. But if you're building agent systems, there are practical steps you can take now.

Monitor input distributions, not just output quality. If the volume or composition of queries hitting an agent shifts significantly, that's a signal — even if no individual query looks wrong. Track category distributions, query length distributions, and topic clustering over time.
Treat upstream agent upgrades as deployment events for your entire chain. When you upgrade one agent, re-run evals on every downstream agent. Don't just test the agent you changed. Test the system.
Build recalibration into your ops process. Have a runbook for "upstream distribution changed." Know in advance which few-shot examples, thresholds, and fine-tuning data need updating, and how to update them quickly.
Design agents to be robust to distributional shift. If your fraud agent completely falls apart when its input distribution shifts from 90% fraud to 50% fraud, it's too tightly calibrated. Build in headroom. Test against varied distributions, not just the current one.

The Road Ahead

We're at an inflection point. The industry is building hierarchies of agents — within companies and across them — and deploying them into high-stakes workflows without the reliability infrastructure to support composition.

The butterfly effect in networked agents isn't a theoretical concern. It's the inevitable consequence of chaining systems that are each calibrated to their current operating conditions, without accounting for the fact that those conditions are shaped by every other agent in the chain. One upgrade anywhere changes operating conditions everywhere downstream.

We solved software composability once. We'll solve it again. But the first step is recognizing that the failure mode is different this time. It's not crashes and errors. It's a silent distributional shift that degrades the entire system while every individual component reports healthy.

The teams that recognize this early and build for it will ship reliable agent systems. The teams that don't will spend months debugging invisible failures that no single team caused, and no single team can fix.

If you're building multi-agent systems or thinking about agent reliability, I'd like to hear how you're approaching this. The problem is real, the tooling isn't, and it's a gap worth closing together.

How Auditing 10 Bio-AI Repositories Shaped STEM-AI (Trust Evaluator for Medical AI Artifact)

Mon, 30 Mar 2026 12:12:10 +0000

Reading path:
This post is part of a series.
(1) STEM-AI introduction — what the framework is and why we built it
(2) Technical audit report — full findings across 10 repositories
(3) Narrative summary — what those findings actually mean

What Text Could See — and What Code Revealed

In March 2026, we ran STEM-AI against 10 high-visibility open-source bio/medical AI repositories.

The framework did what it was designed to do. It surfaced missing disclaimers, absent CI, weak reproducibility signals, and public-facing governance gaps. Those findings mattered, and the scores were directionally right.

But when we reviewed the audits more carefully, one pattern kept appearing: some of the most consequential failures were not visible in the artifact surface at all. They only became obvious when we looked directly at the code.

This function lives inside a repository presented as an AI-driven drug discovery workflow.

def generate_analogues(self, seed_smiles: str, count: int = 3):
    """
    Mocks a generative model (like REINVENT).
    In a real app, this would call a PyTorch model.
    """
    # Simple string manipulation for demo purposes
    analogues = []
    for i in range(count):
        if "C" in seed_smiles:
            new_smi = seed_smiles.replace("C", "C(C)", 1) if i == 0 else seed_smiles + "F"
            analogues.append(new_smi)
        else:
            analogues.append(seed_smiles + "C")
    return analogues

It does not generate molecules. It appends characters.

SMILES (Simplified Molecular Input Line Entry System) is a strict notation for molecular structure. A valid SMILES string encodes real geometry and bonding. Appending C produces a syntactically valid string that represents no real compound. The function runs without error, returns a list, and the pipeline continues downstream.

Our framework scored this repository T0. Correctly. But not because it saw this function.

It scored T0 because the README was missing disclaimers. The CI was absent. Reproducibility was undocumented. Text-path evaluation is designed to measure exactly that. It did.

The audit result was correct. The evidence surface had room to go deeper.

Running the audits showed us what code-path evaluation could add on top.

What Code Access Makes Visible

The drug discovery example was not unusual.

CellAgent's pipeline ends with this call:

# main.py, line 153
final_result = planner.generate_final_result(results)

The method exists. Its body is pass. The pipeline completes without error and produces nothing. A text audit reading the README would have no way to know this.

BioAgents includes a rate limiter for external API calls:

// rateLimiter.ts, lines 62-68
if (process.env.USE_JOB_QUEUE === 'true') {
    await this.rateLimiter.consume(userId);
} else {
    // Job queue disabled - skip rate limiting for direct calls
    return next();
}

USE_JOB_QUEUE defaults to false in .env.example. Every default deployment has rate limiting disabled. The function name implies protection. In default operation, there is none.

The pattern across all three:

the code looks governed.
the behavior tells a different story.
That story is only visible when you read the code.

Text scores and code behavior can diverge. Knowing where and how they diverge is the next layer of evidence worth capturing.

Four Directions the Audits Opened

Reviewing all ten audits, we identified four areas where code-path evaluation could extend what text auditing already does well.

Direction 1: Clinical exposure is visible in imports, not just in README text.

A repository importing pharmacogenomics allele tables has clinical exposure regardless of what its README says. Detecting that dependency at the import level — rather than waiting for a disclaimer — lets the framework flag exposure earlier.

The key distinction is severity: a direct pharmacogenomics import (CYP2D6, CPIC) signals live patient-facing risk and is classified CA-DIRECT.

A general-purpose medical imaging library like pydicom or MONAI is classified CA-INDIRECT — research-use exposure, not necessarily a live clinical output path. The import alone does not determine clinical risk; the classification tier does.

Direction 2: Not all clinical proximity is the same.

A live pharmacogenomics dosage tool and a README roadmap note about a future ClinVar integration are not equivalent risks. Differentiating them — live output vs. research context vs. planned feature — makes the evaluation more precise and makes the accountability expectations more appropriate.

Direction 3: Scoring stability is worth measuring directly.

We ran Stage 1 on one repository in multiple passes. The results ranged across 28 points on the same input. Overlapping trigger conditions between hype-detection items are one contributing factor.

LLM runtime stochasticity is another — the exact split between the two is still under measurement. Adding explicit discrimination examples — what exact phrasing triggers each item, what does not — makes the scoring surface cleaner and reduces the most obvious sources of variance.

Direction 4: Code-path behavior deserves its own scan layer.

A fail-open pattern is a control path that appears to enforce a constraint but defaults to bypassing it. The BioAgents rate limiter above is the example. In a clinical output path, a silent pass-through is not graceful degradation. It is an untraced result that looks like a real one. Building a dedicated scan for these patterns adds a check that text auditing was never meant to provide.

These four directions came directly from running the audits. The scores across the 10 repositories remain as published. Code-path evaluation is what the framework can now add on top of them.

What v1.0.6 Added — Carried Forward in v1.1.2

These changes were introduced in v1.0.6 and are carried forward in the current internal v1.1.2 package. They extend the framework's evidence surface into code-level behavior. Calibration is ongoing.

1. Two Evidence Paths, Not One

We narrowed one of the biggest divergence points by splitting evaluation into a text path and a code path.

The text path works as before: read the README, CHANGELOG, and public posts, score against the rubric. Always available regardless of access to the repository.
The code path activates when the audit has a local clone. It runs through Claude Code, Codex CLI, Gemini CLI, or Copilot CLI. Claims are not interpreted. They are measured. A README that says "IRB-approved data" earns no points for the statement. Points require a provenance artifact in the code.

When code confirms the README, that is a positive signal. When it contradicts it, that contradiction is the finding.

2. Clinical Dependency Detection

At the start of every local audit, a scan script reads Python imports and README keywords. It classifies the result into one of three severity levels:

# ca_detection_scan.sh -- pharmacogenomics section

# CA-DIRECT: live patient-facing output risk
check_import "CPIC|cpic|PharmGx|pharmacogenomic" \\
  "Pharmacogenomics (CPIC/PharmGx)" "CA-DIRECT"

check_import "DPYD|CYP2D6|CYP2C19|allele" \\
  "Pharmacogene alleles" "CA-DIRECT"

# CA-INDIRECT: research-use clinical exposure
check_import "import pydicom|from pydicom" \\
  "pydicom (DICOM imaging)" "CA-INDIRECT"

check_import "import monai|from monai" \\
  "MONAI (medical AI)" "CA-INDIRECT"

Accountability requirements follow the actual clinical proximity of the code. Not the aspirational proximity of the roadmap. A roadmap mention without active implementation is treated as CA-PLANNED rather than collapsed into the same bucket as live clinical output.

The pattern matching is against import statements and function names, not comment text. False positive calibration is still in progress.

3. Code Integrity Scanning (C1-C4)

A second scan handles four code-level checks:

hardcoded credentials (C1)
unpinned dependencies (C2)
clinical-path stubs (C3)
fail-open exception handlers (C4).

The C4 check targets the BioAgents-style pattern. Searching for clinical keywords on the except: line misses most real cases. Clinical context lives in function names and surrounding code. The scan uses a two-pass approach: first identify files with clinical-domain context, then find silent exception handlers within those files.

A silent except: pass in a clinical-context file is a trust-surface failure. The scan makes it visible without requiring a reviewer to read every exception block manually.

4. Discrimination Examples

To reduce the 28-point variance, we added explicit examples for each hype-detection item: what exact phrasing triggers it, what does not, and what the documented edge cases are.

The goal is to reduce obvious scoring drift enough that the same repository is no longer interpreted as two different trust surfaces across different auditors or different LLMs. That goal is not yet verified. The discrimination examples are the primary mechanism toward it.

The Three Questions Now Have a Fourth

In the first post, we described three questions the framework was built around:

Did the repository describe its limits honestly?Did public communication remain consistent with those limits?Did the codebase show evidence of maintenance and biological responsibility?

Running 10 real audits pointed toward a fourth:

4. Does the code actually do what the documentation says — and where it diverges, is that divergence visible and traceable?

That fourth question is what the audit outputs kept surfacing. A function name that sounds real. A pipeline that looks complete. An output that is plausible. An implementation that is a stub, or a control path that silently bypasses its own constraint.

The first three questions can be answered by reading. The fourth requires looking at the code.

What the Framework Added — and What Stays the Same

The first post ended with this:

"STEM-AI is meant to support serious review, not replace it."

That has not changed. Every report carries a non-removable disclaimer: LLM-generated audit, not a regulatory determination, not clinical certification. Every report carries an expiry date. The minimum threshold for supervised pilot consideration is still T3. None of the March 2026 repositories reached it.

What the audits added is narrower — broader evidence coverage on top of an already-working foundation:

A verifiable artifact shifts the accountability surface — it does not eliminate the possibility of falsification. The framework treats its presence as a necessary condition, not a sufficient one.

What the framework gained is that the evidence it counts now extends beyond what authors say about their own code.

Three Directions Still Ahead

Automated re-audit on repository changes. A score from three months ago may not describe the same repository. The trajectory signal measures issue close rate and release frequency across consecutive 90-day windows. It is a partial answer. A CI-triggered re-audit path is the logical next step.

The denominator problem. Zero of 10 repositories reached T3. This may accurately describe the ecosystem's current state. It may also reflect calibration issues in the upper tiers. Distinguishing between the two requires before-and-after auditing of repositories that have received systematic governance remediation.

The Stage 2 redistribution question. Most audits have no cross-platform consistency data. When that data is unavailable, the framework redistributes Stage 2's weight equally between documentation quality and engineering accountability.

For repositories with clinical-direct exposure, a well-written README can then compensate for weak code accountability. A guardrail flags this condition. The current redistribution rule is explicit but not yet final — it remains one of the framework's open calibration questions.

If there are open-source bio-AI repositories you think should be >audited next, drop them in the comments. Bonus if they claim clinical >relevance, drug discovery, or medical reasoning.

STEM-AI v1.1.2 — Trust Evaluation Framework for Medical AI Repositories.

"Code works. But does the author care about the patient?"

# The Pull Request Illusion

Sun, 29 Mar 2026 10:03:29 +0000

Opening

Pull requests used to transfer understanding. Under AI-scale code generation, they increasingly transfer risk.

This article is about how that happened.

For years, the software industry treated the pull request as a workflow object — a diff, a review thread, a merge button, a place where code was inspected before entering the shared system. That description was never wrong. It was just incomplete.

A pull request was not merely a mechanism for checking code. It was a mechanism for converting private reasoning into shared responsibility.

When a developer opened a pull request, they were implicitly saying: I understand this change. I know why it exists. I can explain the tradeoffs. If this breaks in six months, I will not disappear when someone asks why this was written this way.

When a reviewer approved it, they were implicitly saying: I am not just approving syntax. I am accepting the judgment behind this change into a system other people will now have to maintain, debug, secure, and extend.

The pull request was therefore never just a technical tool. It was a social contract around understanding.

And that contract worked — not perfectly, but well enough — as long as the cost of producing code and the cost of understanding code were roughly aligned. Writing code took effort. Structuring a change took time. Opening a pull request usually meant the author had already invested enough thinking that the reviewer was evaluating finished reasoning rather than unfinished guesses.

AI changed that balance.

It made it dramatically cheaper to produce code-like artifacts, but it did not make it equally cheaper to produce shared understanding around them. The result is not simply faster development. The result is that more pull requests now arrive carrying code whose production cost has fallen faster than its comprehension cost.

That is why the pull request is quietly changing roles.

It used to be a mechanism for transferring understanding. Increasingly, it is becoming a mechanism for transferring risk.

Everything else in this article follows from that shift.

What Pull Requests Were Actually For

Pull requests were never primarily about catching syntax errors. Compilers do that better. They were never primarily about style enforcement. Linters do that better. They were never even primarily about tests. CI systems do that better.

Pull requests existed to answer a different question:

Should this decision become part of the system?

A pull request review was historically a review of judgment, not just implementation. Reviewers asked questions like:

Does this belong in this system?
Is this the right abstraction?
What breaks if this assumption changes?
Does this introduce hidden coupling?
Will someone understand this in a year?

These are not syntax questions. They are system questions.

Pull requests worked because code carried the author's reasoning with it. The reviewer was not reconstructing intent from scratch; they were validating intent that already existed.

The process only works if understanding travels with the code.

AI Changed the Economics of Code

AI did not just make coding faster. It changed the cost structure of software development.

Producing code became cheaper.
Producing explanations did not become equally cheaper.
Producing architectural judgment did not become cheaper.
Understanding large changes did not become cheaper.

So the bottleneck moved.

The slowest step in software development used to be writing code. Increasingly, the slowest step is understanding code.

When generation becomes cheap but understanding remains expensive, systems accumulate code faster than they accumulate understanding.

This is the core asymmetry introduced by AI-assisted development.

And pull requests sit exactly at the boundary where code production meets shared responsibility.

So the pull request absorbs the asymmetry.

Why Surface Signals Fool Review

AI-generated code often looks correct before it is understood.

It compiles. It passes tests. It follows style conventions. Variable names look reasonable. The structure resembles patterns the team already uses.

These are surface signals. They are visible and easy to evaluate.

But many real software failures do not come from syntax errors or obvious logic mistakes. They come from deeper system issues:

Hidden state coupling
Incorrect assumptions about invariants
Performance complexity explosions
Concurrency issues
Security boundary violations
Incorrect abstractions that become impossible to change later

These are not visible in a diff unless the reviewer reconstructs the reasoning behind the change.

If the reviewer does not have access to that reasoning, they are not reviewing a decision. They are reviewing an artifact.

And artifacts are much easier to approve than decisions.

The Pull Request Becomes a Risk Transfer Mechanism

When a pull request no longer reliably transfers understanding, it starts transferring something else: responsibility without full context.

The author says the model suggested it. The reviewer sees that tests pass. The change is merged. Months later, a production issue appears, and the question is no longer who wrote the code, but who owns the decision that produced it.

If no one can explain why the code exists in its current form, ownership has already failed.

The pull request has then functioned not as a knowledge transfer mechanism but as a risk transfer mechanism — moving long-term system risk from the moment of creation to the future maintainer.

And once risk is transferred this way, the next question is unavoidable: where did the missing reasoning go?

The Wall of Missing Information

AI systems are very good at filling gaps. When they do not know something, they do not stop. They produce plausible code based on patterns they have seen before.

This creates what can be called the Wall of Missing Information.

On one side of the wall is the code that exists. On the other side are the things the model did not know:

Business constraints
Security assumptions
Performance requirements
Edge cases not in training data
Historical reasons certain patterns exist
Previous failures that shaped the current architecture

The dangerous part is that the generated code does not look incomplete. It looks finished. The absence of information is hidden behind plausibility.

A gate only works if uncertainty remains legible.

If a pull request process cannot distinguish between what is known, what is assumed, and what is unknown, then review becomes approval of plausibility rather than evaluation of understanding.

Code Archaeology

When code is written without durable reasoning attached to it, maintenance turns into archaeology.

Developers encounter a piece of code that works but whose purpose is unclear. Comments are minimal. The original author may have left. The change history shows a pull request, but the reasoning is not documented in a way that explains the decision.

So developers start reverse-engineering intent from behavior. They run experiments. They search old tickets. They try to infer why something was written the way it was.

This is software archaeology: reconstructing decisions from artifacts instead of reading the decisions themselves.

AI-assisted development risks increasing the amount of code that arrives in systems without durable reasoning attached to it, which means future developers spend more time excavating intent rather than extending systems.

And o## What Governance Actually Requiresad of architectural mistakes that look reasonable enough to pass review.

The assembly begins to train on the assembly.

What Governance Actually Requires

If pull requests are increasingly transferring risk rather than understanding, the solution is not simply stricter reviews or better tests. The solution is governance — systems that preserve reasoning, uncertainty, ownership, and traceability.

Each governance requirement exists to close a specific failure mode described earlier.

1. Pull requests must explain why the change exists, not just what changed.
This addresses the surface-signal problem. Code that compiles and passes tests can still be architecturally wrong. Requiring reasoning forces the review to evaluate decisions, not just artifacts.

2. Unknowns and assumptions must be explicitly documented.
This directly addresses the Wall of Missing Information. AI often hides missing context behind plausible code. Declaring unknowns keeps uncertainty visible so review can evaluate risk instead of plausibility.

3. Architectural decisions must have clear ownership.
This addresses risk transfer. If a change alters system behavior, someone must own the decision, not just the diff. Ownership ensures responsibility does not disappear after merge.

4. Decisions, discussions, and code must be traceable to each other.
This addresses code archaeology. When reasoning disappears, future developers reverse‑engineer intent from behavior. Traceability preserves the decision so future engineers do not have to excavate it.

5. Review must evaluate system impact, not just diff correctness.
This addresses hidden system failures — performance, coupling, security boundaries, and long‑term maintainability — which are rarely visible from the diff alone.

Traceability is the minimum requirement for ownership.

If no one can trace why a piece of code exists, then no one truly owns that code, regardless of who merged the pull request.

Conclusion

Pull requests were designed for a world where merging code meant merging understanding.

AI made it possible to merge code without merging th

S&OP Engineering II: Demand Planning from Guessing to Probability

Sat, 28 Mar 2026 16:18:34 +0000

Your Excel says "we'll sell 100 units." A round, clean, deterministic number. What if you sell 120? Stockout, unhappy customer, contractual penalty. What if you sell 50? 50 units sitting in your warehouse, immobilizing capital that could be generating returns.

The problem isn't the forecast itself. It's the arrogance of the single number.

In Chapter 1 we built a "Quality Valve" that filters ERP noise. Now that we have a pure signal, we're going to do something Excel can't: measure uncertainty.

Executive Summary: A probabilistic forecast doesn't tell you "you'll sell 100." It tells you "with 95% probability, you'll sell between 35 and 157." That uncertainty band is the mathematical foundation for calculating your Safety Stock without resorting to rules of thumb.

The Architecture: MLOps, Not Scripts

We haven't written a throwaway script. We've connected our mathematical brain (Python/Prophet) directly to our Single Source of Truth (Supabase/PostgreSQL).

The key design is the demand_forecasts table. Notice one column that separates this from a basic tutorial:

CREATE TABLE demand_forecasts (
    id              UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    execution_date  DATE NOT NULL,     -- ← When we ran the model
    ds              DATE NOT NULL,     -- Predicted future date
    yhat            NUMERIC NOT NULL,  -- Central prediction
    yhat_lower      NUMERIC NOT NULL,  -- Lower bound (Safety Stock)
    yhat_upper      NUMERIC NOT NULL,  -- Upper bound (Risk)
    model_version   TEXT NOT NULL,     -- Traceability
    UNIQUE(execution_date, ds)         -- One forecast per execution and date
);

Why execution_date? Because in 6 months, when you want to audit whether your model was accurate, you need to know when you made the prediction versus what actually happened. This is what MLOps calls Snapshotting: recording the context of each execution to evaluate model drift over time.

Without this column, you have a model. With it, you have an auditable system.

The Engineering: Why Prophet (and Not a Textbook ARIMA)

Prophet is a time series engine developed by Meta, designed for irregular business data: gaps, holidays, trend changes. Exactly what a real supply chain has.

Here's the core fragment from our ProphetPredictor class:

def train_model(self, country_code='ES'):
    """
    Trains Prophet with two key S&OP configurations:
    - interval_width: Confidence interval width
    - country_holidays: Country operational context
    """
    self.model = Prophet(
        interval_width=0.95,     # ← 95% confidence interval
        weekly_seasonality=True,
        yearly_seasonality=True
    )
    # Holidays that alter loading/unloading patterns
    self.model.add_country_holidays(country_name=country_code)
    self.model.fit(self.ts_df)

Two engineering decisions that separate us from a YouTube tutorial:

interval_width=0.95: This is not a decorative parameter. The upper bound (yhat_upper) represents the maximum probable demand with 95% confidence. This is literally your calculation basis for Safety Stock: Safety Stock = yhat_upper - yhat. Without this interval, your safety stock is a gut feeling; with it, it's mathematics.

add_country_holidays('ES'): In S&OP, holidays aren't "days off." They're operational anomalies: the factory closes, the warehouse doesn't dispatch, transportation stops. If the model doesn't know that August 15th is a national holiday in Spain, it will interpret the drop in orders as a downward trend. This corrupts the September prediction.

Translating Math into Business Decisions

Theory is nice. But a CEO doesn't approve budgets based on Python code. They need visual evidence.

The Probabilistic Forecast

The black dots are your actual historical demand. The central blue line is the prediction (yhat). The shaded band is the 95% confidence interval. The greater the historical volatility, the wider the band → the more Safety Stock you need → the more cash you immobilize. This band is the conversation you should be having with your CFO.

Explainability: Separating Trend from Seasonal Noise

This is XAI (Explainable AI) applied to business. The model separates Trend (is the business growing or shrinking?) from Seasonality (are there spikes by time of year?) and holiday effects. This is what the CEO needs to see to approve the operations plan: knowing whether growth is real or just a calendar effect.

Open Kitchen: Try It Yourself

I distrust articles I can't execute. That's why I've prepared an isolated environment where you can train the model on an anonymized snapshot of the data we just cleaned in Chapter 1.

You don't need to install Python, Prophet, or configure Supabase. You just need a browser:

Open the interactive Notebook on Google Colab

Hit "Play" on the cells and watch a probabilistic forecast with uncertainty bands being generated. Modify the horizon, the country, the confidence interval. Do engineering, not faith.

Next Step: From Prediction to Decision

Now we know the probability of what we're going to sell. The question shifts from "how much will we sell?" to "what should we manufacture or purchase to maximize margin and minimize risk?".

In Chapter 3, we'll introduce Mathematical Optimization (PuLP) and Theory of Constraints to transform predictions into supply decisions: how much to buy, when to produce, and how to distribute finite resources while minimizing costs.

The difference between an Operations Director who reacts and one who decides is a mathematical model between the data and the action.

Breaking the AI Data Bottleneck: How Hammerspace's AI Data Platform Eliminates Migration Nightmares

Tue, 17 Mar 2026 00:00:00 +0000

If you're an engineer or architect trying to move AI projects from pilot to production, you've likely hit the same wall: data preparation is eating your budget, timeline, and sanity. Between manually curating datasets, managing governance policies, and migrating terabytes of enterprise data into yet another specialized storage system, many AI initiatives stall before they can prove ROI.

Hammerspace's newly available AI Data Platform (AIDP) tackles this problem with a fundamentally different approach: use your data where it already lives. Built on NVIDIA's reference design and validated on Cisco UCS infrastructure, this turnkey solution eliminates the data migration nightmare while providing the performance and governance enterprises need for production AI.

The Problem: AI Projects Drowning in Data Preparation

Enterprise AI faces a brutal reality: most projects fail not because of model quality, but because of data complexity. Here's what typically happens:

Data Fragmentation: Your enterprise data lives across NetApp filers, AWS S3 buckets, Azure Blob storage, on-premises object stores, and various departmental shares
Manual Curation: Teams spend months manually identifying, copying, and organizing relevant data
Tool Sprawl: You're juggling 15+ different tools for discovery, cataloging, classification, governance, and movement
Migration Hell: To get started with AI, you need to copy everything into a new AI-specific storage system
Governance Nightmares: Each copy creates new security, compliance, and data sovereignty issues

Warning: The traditional approach of building a separate AI data estate can add months to your timeline and millions to your budget - and you haven't even started training models yet.

How Data-in-Place Architecture Actually Works

The core innovation of Hammerspace's platform is data assimilation - the ability to create a unified logical view of distributed data without physical migration. Here's the technical architecture:

Global Namespace Layer

Traditional Approach:
[NetApp Filer] → COPY → [AI Storage System]
[AWS S3]       → COPY → [AI Storage System]
[Azure Blob]   → COPY → [AI Storage System]

Hammerspace Approach:
[NetApp Filer] ─┐
[AWS S3]       ─┼─→ [Global Namespace] → [AI Workloads]
[Azure Blob]   ─┘

The global namespace provides a single mount point that transparently accesses data across heterogeneous storage systems. You point at existing shares and start querying immediately - no migration required.

Model Context Protocol (MCP) Server Integration

At the heart of the platform is Hammerspace's MCP server, which handles intelligent data discovery and orchestration:

# Natural language query interface
# Developer or data scientist can query directly without IT involvement

query = "Find all customer transaction data from Q4 2025 
         with PII redacted for model training"

# MCP server coordinates with:
# 1. Secuvy DSPM for governance validation
# 2. Hammerspace namespace for data location
# 3. NVIDIA NIM microservices for processing
# 4. GPU resources for initial data processing

The MCP server provides full transparency into what's happening under the hood. Developers and data scientists can see:

Which storage systems are being queried
What governance policies are being applied
Where bottlenecks exist in the pipeline
Which stages are taking 100ms vs 1ms

Note: This isn't a black box - the natural language interface shows you the generated queries and lets you refine them based on what you see happening in real-time.

Step 1: Architecture Planning - Understanding the Complete Stack

The AI Data Platform ships as an integrated hardware/software solution on Cisco UCS infrastructure:

Hardware Components:
├── Cisco UCS Servers
├── NVIDIA RTX GPUs (for local data processing)
├── Configurable capacity based on workload
└── Network infrastructure

Software Stack:
├── NVIDIA AI Enterprise Software
│   ├── NIM Microservices
│   ├── NeMo Retriever
│   └── NVIDIA AI Data Platform reference design
├── Hammerspace Global Data Platform
│   ├── MCP Server
│   ├── Data Assimilation Engine
│   └── Orchestration Layer
└── Secuvy DSPM (Data Security Posture Management)
    ├── Governance Policy Engine
    ├── Compliance Monitoring
    └── Security Posture Management

Why Local GPUs Matter

The inclusion of NVIDIA RTX GPUs in the data platform itself is strategically important:

# Traditional workflow - all processing on expensive training GPUs
data = load_from_storage()  # Uses H100/H200 GPU cycles
cleaned = clean_data(data)  # Uses H100/H200 GPU cycles
prepared = prepare_for_training(data)  # Uses H100/H200 GPU cycles
train_model(prepared)  # Finally using GPUs for actual training

# Hammerspace workflow - preprocessing on local RTX GPUs
data = query_via_mcp("customer_data Q4 2025")  # RTX GPU processing
# Data arrives pre-processed and governance-validated
train_model(data)  # H100/H200 GPUs only used for actual training

This architecture keeps expensive training GPUs focused on training, while local RTX GPUs handle data preparation, governance validation, and initial processing.

Step 2: Integration with Existing Infrastructure

Using Data in Place Without Migration

Here's how the platform integrates with your existing storage:

# Hammerspace configuration example
data_sources:
  - name: "netapp_production"
    type: "nfs"
    mount: "netapp01.company.com:/vol/production"
    governance_tier: "tier1"
    
  - name: "aws_archive"
    type: "s3"
    bucket: "s3://company-archive-us-east-1"
    governance_tier: "tier2"
    
  - name: "azure_backup"
    type: "blob"
    container: "backup-container"
    governance_tier: "tier3"

global_namespace:
  mount_point: "/global/enterprise_data"
  consistency: "strong"
  cache_policy: "intelligent"

When you query data through the MCP server, Hammerspace:

Identifies which storage systems contain relevant data
Validates governance requirements with Secuvy
Streams only the required data to GPU resources
Maintains metadata about data lineage and usage

No Rip-and-Replace for Existing Tools

For teams already using orchestration tools like Apache Airflow, dbt, or workflow managers, Hammerspace is the data orchestrator, not the workflow orchestrator:

# Existing Airflow DAG continues to work
from airflow import DAG
from airflow.operators.python import PythonOperator

def prepare_training_data():
    # Your existing code doesn't change
    # Hammerspace handles data location and movement transparently
    data_path = "/global/enterprise_data/training_set"
    # This path now spans multiple storage systems
    return process_data(data_path)

dag = DAG('ai_training_pipeline')
prep_task = PythonOperator(
    task_id='prepare_data',
    python_callable=prepare_training_data,
    dag=dag
)

The platform works alongside tools like:

Run:ai (NVIDIA's workload orchestration)
Parallel Works (specialized HPC orchestration)
Industry-specific workflow managers

Step 3: Implementation Guide for Production Deployment

Deployment Timeline

Based on validation work with SHI's labs and early customers:

Week 1: Initial Setup
- Cisco UCS hardware arrives pre-configured
- Install Hammerspace software (< 30 minutes)
- Configure connections to existing storage
- Set up governance policies with Secuvy

Week 2: Data Discovery and Testing
- MCP server indexes existing data sources
- Data scientists begin natural language queries
- Validate governance policies are working
- Test data access patterns and performance

Week 3: Pilot Workload
- Run initial AI workload on curated dataset
- Monitor performance metrics
- Identify any bottlenecks in pipeline
- Refine policies based on real usage

Week 4+: Scale to Production
- Expand to additional data sources
- Increase GPU allocation as needed
- Automate data refresh on scheduled cadence
- Deploy to additional business units

Tip: The ability to start with existing infrastructure and data means you can prove ROI before major capital expenditure on new storage systems.

Handling the SSD Crisis and GPU Availability

The current shortage of SSDs and memory creates unique challenges. Hammerspace's approach provides flexibility:

Scenario 1: Can't get SSDs for new storage
✓ Use existing storage infrastructure
✓ Add Hammerspace orchestration layer
✓ Deploy GPUs where hardware IS available

Scenario 2: Need flexibility in GPU selection
✓ Not locked into H200 if unavailable
✓ Can use RTX GPUs for data processing
✓ Burst to cloud GPUs when needed

Scenario 3: Multi-site deployment required
✓ Central data platform management
✓ GPUs deployed at edge locations
✓ Data orchestrated based on proximity

Advanced Features: Automated Data Refresh and Governance

Continuous Data Discovery

One critical difference between pilot and production AI: data isn't static. The platform handles this through automated refresh:

// Automated data refresh configuration
{
  "refresh_schedule": "0 2 * * *",  // Daily at 2 AM
  "scan_locations": [
    "/global/enterprise_data/sales",
    "/global/enterprise_data/customer_service",
    "/global/enterprise_data/products"
  ],
  "governance_validation": {
    "provider": "secuvy",
    "policies": ["pii_redaction", "gdpr_compliance", "data_retention"],
    "auto_tag": true
  },
  "notification": {
    "new_data_found": "slack://ai-team-channel",
    "governance_violations": "email://*Emails are not allowed*"
  }
}

This automated refresh ensures that:

New data is automatically discovered and made available
Governance policies are continuously enforced
Data scientists don't need IT to add new data sources
Compliance teams have visibility into what data is being used

Data Lineage and Audit Trail

For regulated industries, tracking which data trained which models is critical:

-- Example query against Hammerspace metadata
SELECT 
    model_id,
    training_date,
    source_storage_system,
    file_path,
    governance_policy_applied,
    data_classification
FROM hammerspace.training_metadata
WHERE model_id = 'customer_churn_v2'
ORDER BY training_date DESC;

This provides the audit trail required for regulatory compliance while maintaining the flexibility to use data across multiple storage systems.

Real-World Performance Considerations

Time to First Token vs Time to Production

While traditional benchmarks focus on "time to first token," Hammerspace emphasizes total time to production:

Traditional Approach:
- Plan new storage system: 2 weeks
- Procurement and delivery: 4-8 weeks
- Installation and configuration: 2 weeks
- Data migration: 4-12 weeks (depending on volume)
- Governance setup: 2-4 weeks
- Testing and validation: 2 weeks
Total: 16-30 weeks before first inference

Hammerspace Approach:
- Deploy platform on existing infrastructure: 1 week
- Data discovery and indexing: 1 week
- Governance policy configuration: 1 week
- Testing and validation: 1 week
Total: 4 weeks to first inference

Important: This 4-10x reduction in time to production often matters more to enterprises than marginal improvements in inference latency.

Pipeline Bottleneck Identification

The MCP server provides visibility into where bottlenecks exist:

# Example debugging output from MCP server
Pipeline Stage Analysis:
├── Data Discovery: 15ms ✓
├── Governance Validation: 120ms ⚠️
├── Data Retrieval from NetApp: 45ms ✓
├── Format Conversion: 250ms ❌
├── GPU Transfer: 30ms ✓
└── Inference Ready: 460ms total

Recommendation: Format conversion is the bottleneck.
Consider pre-processing this data type or using
GPU-accelerated conversion on RTX resources.

This transparency allows developers to optimize the specific stages causing delays rather than guessing.

Ecosystem Integration: The Complete Solution

Validated Partners

The platform ships with validated integrations:

Cisco UCS: Complete hardware infrastructure
NVIDIA AI Enterprise: NIM microservices, NeMo Retriever, AI software stack
Secuvy DSPM: End-to-end governance and compliance
SHI: Systems integrator providing deployment and support

API-Level Integration

All components integrate at the API level for automation:

# Example: Triggering inference job with automated data prep
import hammerspace_client as hs
import nvidia_nim as nim

# Query for relevant data via MCP
data_query = hs.mcp.query(
    prompt="Customer support tickets mentioning product X in 2025",
    governance_policy="standard_pii_redaction"
)

# Data automatically staged to GPU resources
# Governance validated by Secuvy
# Ready for immediate inference

result = nim.inference(
    model="customer_sentiment_v3",
    data=data_query.staged_location,
    gpu_allocation="auto"
)

Migration Path for Different Developer Profiles

Scenario 1: Team New to AI Infrastructure

Your Situation:
- First AI project in production
- Limited AI infrastructure expertise
- Want turnkey solution

Migration Path:
✓ Deploy complete Cisco UCS bundle
✓ Use natural language MCP interface
✓ Let platform handle all orchestration
✓ Work with SHI for deployment support

Scenario 2: Team with Existing AI Tools

Your Situation:
- Already using Airflow, MLflow, etc.
- Have existing data pipelines
- Want better data access without rewriting everything

Migration Path:
✓ Deploy Hammerspace as data layer
✓ Keep existing workflow orchestration
✓ Point tools at global namespace
✓ Gradually adopt MCP for new projects

Scenario 3: Multi-Cloud Enterprise

Your Situation:
- Data across AWS, Azure, on-prem
- Need data sovereignty compliance
- Want flexibility to use GPUs wherever available

Migration Path:
✓ Deploy Hammerspace data assimilation
✓ Create global namespace across all locations
✓ Use Secuvy for sovereignty policies
✓ Orchestrate GPUs based on data location and cost

Overcoming Data Sprawl and Security Concerns

Every data copy creates exponentially more security risk:

Traditional Approach:
Original Data (NetApp)
    └─> Copy 1 (AI Storage for Training)
        └─> Copy 2 (Dev/Test Environment)
            └─> Copy 3 (Cloud Backup)
                └─> Copy 4 (Partner Access)

Each copy needs:
- Separate governance policies
- Individual security monitoring
- Distinct compliance tracking
- Separate backup/disaster recovery

Result: 5x the attack surface, 5x the compliance burden

Hammerspace's data-in-place approach:

Single Data Source (in original location)
    └─> Global Namespace (logical view only)
        └─> Secuvy DSPM (single governance point)
            └─> Orchestrated access (policy-driven)

Result: 1x attack surface, centralized governance

Conclusion

The enterprise AI bottleneck isn't model quality or GPU shortage - it's data preparation complexity. Hammerspace's AI Data Platform addresses this by fundamentally changing the approach: instead of forcing enterprises to migrate data into yet another specialized system, it brings AI capabilities to where data already lives.

For developers and architects, this means:

Faster time to production (weeks instead of months)
Lower infrastructure costs (use existing storage instead of buying new)
Reduced complexity (one orchestration layer instead of 15 tools)
Better governance (centralized policies instead of per-copy management)
Greater flexibility (deploy GPUs where available, access data anywhere)

The platform's integration of NVIDIA's reference design, Cisco UCS infrastructure, Secuvy governance, and Hammerspace orchestration provides a turnkey solution that addresses the complete data-to-inference pipeline. And critically, it does this while working with your existing tools and workflows rather than requiring a rip-and-replace approach.

As enterprises move from AI pilots to production deployments, the ability to scale without rebuilding your entire data infrastructure becomes the difference between success and stalled initiatives. The AI Data Platform's data-in-place architecture, automated governance, and transparent orchestration make that transition possible.

FAQ

How does this handle data consistency when accessing multiple storage systems?

The global namespace maintains strong consistency through distributed metadata management. When data is accessed, Hammerspace coordinates with the source storage system to ensure you always see the current version. For write operations, policies determine whether changes propagate back to source systems or are staged separately.

What happens if one of my source storage systems goes offline?

The platform includes intelligent caching and redundancy. Frequently accessed data can be cached in the Hammerspace tier, and governance policies can specify which data should have redundant copies for availability. The MCP server will route queries to available systems and alert you to unavailable sources.

Can I use this with my existing H100 GPU cluster?

Absolutely. The Cisco UCS bundle is one deployment option, but Hammerspace software can integrate with existing GPU infrastructure. The platform provides the data orchestration layer while your existing compute resources handle training and inference. This is particularly valuable for teams that already have GPU allocations.

How does natural language querying actually work for data scientists?

The MCP server uses NVIDIA NIM microservices to interpret natural language queries and translate them into specific data operations. Data scientists can query like "Find all customer transaction data from Q4 with PII redacted" and see the underlying operations being performed. They can refine queries based on results and save frequently used queries as templates.

What's the learning curve for developers already using tools like Airflow?

Minimal. Your existing code continues to work - you're simply pointing at a different data path (the global namespace mount point). For new projects, you can choose to use the MCP server's natural language interface or continue with your familiar tools. Most teams adopt a hybrid approach: existing workflows stay unchanged while new AI projects use the modern interfaces.

Breaking the AI Data Bottleneck: How Hammerspace's AI Data Platform Eliminates the Migration Nightma

Mon, 16 Mar 2026 21:24:06 +0000

The Problem: AI Projects Drowning in Data Preparation

Enterprise AI faces a brutal reality: most projects fail not because of model quality, but because of data complexity. Here's what typically happens:

Data Fragmentation: Your enterprise data lives across NetApp filers, AWS S3 buckets, Azure Blob storage, on-premises object stores, and various departmental shares
Manual Curation: Teams spend months manually identifying, copying, and organizing relevant data
Tool Sprawl: You're juggling 15+ different tools for discovery, cataloging, classification, governance, and movement
Migration Hell: To get started with AI, you need to copy everything into a new AI-specific storage system
Governance Nightmares: Each copy creates new security, compliance, and data sovereignty issues

Warning: The traditional approach of building a separate AI data estate can add months to your timeline and millions to your budget - and you haven't even started training models yet.

How Data-in-Place Architecture Actually Works

Global Namespace Layer

Traditional Approach:
[NetApp Filer] → COPY → [AI Storage System]
[AWS S3]       → COPY → [AI Storage System]
[Azure Blob]   → COPY → [AI Storage System]

Hammerspace Approach:
[NetApp Filer] ─┐
[AWS S3]       ─┼─→ [Global Namespace] → [AI Workloads]
[Azure Blob]   ─┘

Model Context Protocol (MCP) Server Integration

At the heart of the platform is Hammerspace's MCP server, which handles intelligent data discovery and orchestration:

# Natural language query interface
# Developer or data scientist can query directly without IT involvement

query = "Find all customer transaction data from Q4 2025 
         with PII redacted for model training"

# MCP server coordinates with:
# 1. Secuvy DSPM for governance validation
# 2. Hammerspace namespace for data location
# 3. NVIDIA NIM microservices for processing
# 4. GPU resources for initial data processing

The MCP server provides full transparency into what's happening under the hood. Developers and data scientists can see:

Which storage systems are being queried
What governance policies are being applied
Where bottlenecks exist in the pipeline
Which stages are taking 100ms vs 1ms

Note: This isn't a black box - the natural language interface shows you the generated queries and lets you refine them based on what you see happening in real-time.

Step 1: Architecture Planning - Understanding the Complete Stack

The AI Data Platform ships as an integrated hardware/software solution on Cisco UCS infrastructure:

Hardware Components:
├── Cisco UCS Servers
├── NVIDIA RTX GPUs (for local data processing)
├── Configurable capacity based on workload
└── Network infrastructure

Software Stack:
├── NVIDIA AI Enterprise Software
│   ├── NIM Microservices
│   ├── NeMo Retriever
│   └── NVIDIA AI Data Platform reference design
├── Hammerspace Global Data Platform
│   ├── MCP Server
│   ├── Data Assimilation Engine
│   └── Orchestration Layer
└── Secuvy DSPM (Data Security Posture Management)
    ├── Governance Policy Engine
    ├── Compliance Monitoring
    └── Security Posture Management

Why Local GPUs Matter

The inclusion of NVIDIA RTX GPUs in the data platform itself is strategically important:

# Traditional workflow - all processing on expensive training GPUs
data = load_from_storage()  # Uses H100/H200 GPU cycles
cleaned = clean_data(data)  # Uses H100/H200 GPU cycles
prepared = prepare_for_training(data)  # Uses H100/H200 GPU cycles
train_model(prepared)  # Finally using GPUs for actual training

# Hammerspace workflow - preprocessing on local RTX GPUs
data = query_via_mcp("customer_data Q4 2025")  # RTX GPU processing
# Data arrives pre-processed and governance-validated
train_model(data)  # H100/H200 GPUs only used for actual training

This architecture keeps expensive training GPUs focused on training, while local RTX GPUs handle data preparation, governance validation, and initial processing.

Step 2: Integration with Existing Infrastructure

Using Data in Place Without Migration

Here's how the platform integrates with your existing storage:

# Hammerspace configuration example
data_sources:
  - name: "netapp_production"
    type: "nfs"
    mount: "netapp01.company.com:/vol/production"
    governance_tier: "tier1"
    
  - name: "aws_archive"
    type: "s3"
    bucket: "s3://company-archive-us-east-1"
    governance_tier: "tier2"
    
  - name: "azure_backup"
    type: "blob"
    container: "backup-container"
    governance_tier: "tier3"

global_namespace:
  mount_point: "/global/enterprise_data"
  consistency: "strong"
  cache_policy: "intelligent"

When you query data through the MCP server, Hammerspace:

Identifies which storage systems contain relevant data
Validates governance requirements with Secuvy
Streams only the required data to GPU resources
Maintains metadata about data lineage and usage

No Rip-and-Replace for Existing Tools

For teams already using orchestration tools like Apache Airflow, dbt, or workflow managers, Hammerspace is the data orchestrator, not the workflow orchestrator:

# Existing Airflow DAG continues to work
from airflow import DAG
from airflow.operators.python import PythonOperator

def prepare_training_data():
    # Your existing code doesn't change
    # Hammerspace handles data location and movement transparently
    data_path = "/global/enterprise_data/training_set"
    # This path now spans multiple storage systems
    return process_data(data_path)

dag = DAG('ai_training_pipeline')
prep_task = PythonOperator(
    task_id='prepare_data',
    python_callable=prepare_training_data,
    dag=dag
)

The platform works alongside tools like:

Run:ai (NVIDIA's workload orchestration)
Parallel Works (specialized HPC orchestration)
Industry-specific workflow managers

Step 3: Implementation Guide for Production Deployment

Deployment Timeline

Based on validation work with SHI's labs and early customers:

Week 1: Initial Setup
- Cisco UCS hardware arrives pre-configured
- Install Hammerspace software (< 30 minutes)
- Configure connections to existing storage
- Set up governance policies with Secuvy

Week 2: Data Discovery and Testing
- MCP server indexes existing data sources
- Data scientists begin natural language queries
- Validate governance policies are working
- Test data access patterns and performance

Week 3: Pilot Workload
- Run initial AI workload on curated dataset
- Monitor performance metrics
- Identify any bottlenecks in pipeline
- Refine policies based on real usage

Week 4+: Scale to Production
- Expand to additional data sources
- Increase GPU allocation as needed
- Automate data refresh on scheduled cadence
- Deploy to additional business units

Tip: The ability to start with existing infrastructure and data means you can prove ROI before major capital expenditure on new storage systems.

Handling the SSD Crisis and GPU Availability

The current shortage of SSDs and memory creates unique challenges. Hammerspace's approach provides flexibility:

Scenario 1: Can't get SSDs for new storage
✓ Use existing storage infrastructure
✓ Add Hammerspace orchestration layer
✓ Deploy GPUs where hardware IS available

Scenario 2: Need flexibility in GPU selection
✓ Not locked into H200 if unavailable
✓ Can use RTX GPUs for data processing
✓ Burst to cloud GPUs when needed

Scenario 3: Multi-site deployment required
✓ Central data platform management
✓ GPUs deployed at edge locations
✓ Data orchestrated based on proximity

Advanced Features: Automated Data Refresh and Governance

Continuous Data Discovery

One critical difference between pilot and production AI: data isn't static. The platform handles this through automated refresh:

// Automated data refresh configuration
{
  "refresh_schedule": "0 2 * * *",  // Daily at 2 AM
  "scan_locations": [
    "/global/enterprise_data/sales",
    "/global/enterprise_data/customer_service",
    "/global/enterprise_data/products"
  ],
  "governance_validation": {
    "provider": "secuvy",
    "policies": ["pii_redaction", "gdpr_compliance", "data_retention"],
    "auto_tag": true
  },
  "notification": {
    "new_data_found": "slack://ai-team-channel",
    "governance_violations": "email://*Emails are not allowed*"
  }
}

This automated refresh ensures that:

New data is automatically discovered and made available
Governance policies are continuously enforced
Data scientists don't need IT to add new data sources
Compliance teams have visibility into what data is being used

Data Lineage and Audit Trail

For regulated industries, tracking which data trained which models is critical:

-- Example query against Hammerspace metadata
SELECT 
    model_id,
    training_date,
    source_storage_system,
    file_path,
    governance_policy_applied,
    data_classification
FROM hammerspace.training_metadata
WHERE model_id = 'customer_churn_v2'
ORDER BY training_date DESC;

This provides the audit trail required for regulatory compliance while maintaining the flexibility to use data across multiple storage systems.

Real-World Performance Considerations

Time to First Token vs Time to Production

While traditional benchmarks focus on "time to first token," Hammerspace emphasizes total time to production:

Traditional Approach:
- Plan new storage system: 2 weeks
- Procurement and delivery: 4-8 weeks
- Installation and configuration: 2 weeks
- Data migration: 4-12 weeks (depending on volume)
- Governance setup: 2-4 weeks
- Testing and validation: 2 weeks
Total: 16-30 weeks before first inference

Hammerspace Approach:
- Deploy platform on existing infrastructure: 1 week
- Data discovery and indexing: 1 week
- Governance policy configuration: 1 week
- Testing and validation: 1 week
Total: 4 weeks to first inference

Important: This 4-10x reduction in time to production often matters more to enterprises than marginal improvements in inference latency.

Pipeline Bottleneck Identification

The MCP server provides visibility into where bottlenecks exist:

# Example debugging output from MCP server
Pipeline Stage Analysis:
├── Data Discovery: 15ms ✓
├── Governance Validation: 120ms ⚠️
├── Data Retrieval from NetApp: 45ms ✓
├── Format Conversion: 250ms ❌
├── GPU Transfer: 30ms ✓
└── Inference Ready: 460ms total

Recommendation: Format conversion is the bottleneck.
Consider pre-processing this data type or using
GPU-accelerated conversion on RTX resources.

This transparency allows developers to optimize the specific stages causing delays rather than guessing.

Ecosystem Integration: The Complete Solution

Validated Partners

The platform ships with validated integrations:

Cisco UCS: Complete hardware infrastructure
NVIDIA AI Enterprise: NIM microservices, NeMo Retriever, AI software stack
Secuvy DSPM: End-to-end governance and compliance
SHI: Systems integrator providing deployment and support

API-Level Integration

All components integrate at the API level for automation:

# Example: Triggering inference job with automated data prep
import hammerspace_client as hs
import nvidia_nim as nim

# Query for relevant data via MCP
data_query = hs.mcp.query(
    prompt="Customer support tickets mentioning product X in 2025",
    governance_policy="standard_pii_redaction"
)

# Data automatically staged to GPU resources
# Governance validated by Secuvy
# Ready for immediate inference

result = nim.inference(
    model="customer_sentiment_v3",
    data=data_query.staged_location,
    gpu_allocation="auto"
)

Migration Path for Different Developer Profiles

Scenario 1: Team New to AI Infrastructure

Your Situation:
- First AI project in production
- Limited AI infrastructure expertise
- Want turnkey solution

Migration Path:
✓ Deploy complete Cisco UCS bundle
✓ Use natural language MCP interface
✓ Let platform handle all orchestration
✓ Work with SHI for deployment support

Scenario 2: Team with Existing AI Tools

Your Situation:
- Already using Airflow, MLflow, etc.
- Have existing data pipelines
- Want better data access without rewriting everything

Migration Path:
✓ Deploy Hammerspace as data layer
✓ Keep existing workflow orchestration
✓ Point tools at global namespace
✓ Gradually adopt MCP for new projects

Scenario 3: Multi-Cloud Enterprise

Your Situation:
- Data across AWS, Azure, on-prem
- Need data sovereignty compliance
- Want flexibility to use GPUs wherever available

Migration Path:
✓ Deploy Hammerspace data assimilation
✓ Create global namespace across all locations
✓ Use Secuvy for sovereignty policies
✓ Orchestrate GPUs based on data location and cost

Overcoming Data Sprawl and Security Concerns

Every data copy creates exponentially more security risk:

Traditional Approach:
Original Data (NetApp)
    └─> Copy 1 (AI Storage for Training)
        └─> Copy 2 (Dev/Test Environment)
            └─> Copy 3 (Cloud Backup)
                └─> Copy 4 (Partner Access)

Each copy needs:
- Separate governance policies
- Individual security monitoring
- Distinct compliance tracking
- Separate backup/disaster recovery

Result: 5x the attack surface, 5x the compliance burden

Hammerspace's data-in-place approach:

Single Data Source (in original location)
    └─> Global Namespace (logical view only)
        └─> Secuvy DSPM (single governance point)
            └─> Orchestrated access (policy-driven)

Result: 1x attack surface, centralized governance

Conclusion

For developers and architects, this means:

Faster time to production (weeks instead of months)
Lower infrastructure costs (use existing storage instead of buying new)
Reduced complexity (one orchestration layer instead of 15 tools)
Better governance (centralized policies instead of per-copy management)
Greater flexibility (deploy GPUs where available, access data anywhere)

FAQ

How does this handle data consistency when accessing multiple storage systems?

What happens if one of my source storage systems goes offline?

Can I use this with my existing H100 GPU cluster?

How does natural language querying actually work for data scientists?

What's the learning curve for developers already using tools like Airflow?

FuriosaAI and Helikai Partner on Power-Efficient Enterprise AI Stack

Fri, 27 Feb 2026 15:33:09 +0000

A month after presenting their micro AI agent approach at the 66th IT Press Tour, Helikai announced a strategic partnership with FuriosaAI that addresses one of enterprise AI's most overlooked constraints: power consumption. The partnership certifies Helikai's full software stack on FuriosaAI's RNGD inference servers, which deliver 512 TOPS of INT8 compute while consuming just 180 watts—a fraction of comparable GPU-based systems.

For enterprises deploying AI agents in edge data centers, factories, hospitals, or research labs, this power efficiency unlocks deployment scenarios that weren't previously practical. You can now run production-grade AI automation in environments with limited power and cooling capacity, without sacrificing the performance needed for complex multi-agent workflows.

The Hardware Story: RNGD Architecture

FuriosaAI's RNGD servers support up to eight accelerator cards per system, delivering 4 petaFLOPS of compute with 384GB of HBM3 memory. The architecture is specifically optimized for inference workloads rather than training, which aligns perfectly with Helikai's deployment model of pre-built, domain-specific agents.

The platform includes a full SDK with native support for cutting-edge models, including Qwen 3, and implements hybrid batching for multi-turn RAG (Retrieval Augmented Generation) and agentic workloads. This matters because Helikai's agents often operate in conversational modes where maintaining context across multiple turns is critical for workflow automation.

The power story becomes compelling when you consider deployment at scale. A traditional GPU-based inference server might consume 300-400 watts per card. In a data center running dozens of AI workloads, that power differential adds up to significant operational cost savings and reduced cooling requirements. For edge deployments or facilities with power constraints, it's often the difference between feasible and infeasible.

Helikai's Hybrid Architecture on Furiosa

Helikai's platform combines three distinct approaches: Instruction-Tuned Language Models (ILMs), agentic AI, and deterministic logic. This hybrid architecture runs efficiently on RNGD hardware because each component can be optimized differently.

The ILMs handle semantic understanding and generation tasks—reading contracts, extracting structured data from documents, and generating responses to queries. The agentic layer orchestrates multi-step workflows, deciding which specialized agent to invoke next based on intermediate results. The deterministic logic handles operations where AI creativity isn't wanted: database queries, rule validation, compliance checks, and mathematical calculations.

By offloading deterministic operations to the CPU and reserving the AI accelerator for actual inference tasks, Helikai's Mālama optimization layer reduces the overall compute footprint. Token request and response envelopes are minimized, which matters when you're processing thousands of documents per hour or running conversational agents that handle hundreds of concurrent sessions.

The Data Sovereignty Angle

The partnership press release emphasizes "security, privacy, and data sovereignty" as core design principles. This isn't marketing fluff—it addresses the primary barrier to enterprise AI adoption in regulated industries.

Financial services firms, healthcare organizations, legal practices, and insurance companies face strict requirements about where data can be processed and stored. Sending contracts, patient records, or claims data to external API endpoints for processing creates compliance nightmares. The combination of Helikai's on-premise deployment model and Furiosa's inference-optimized hardware enables these organizations to run sophisticated AI automation entirely within their corporate boundaries.

The technical implementation supports this. Helikai's SPRAG (Secure Private Retrieval Augmented Generation) platform can run completely air-gapped, with no internet connectivity required. All models, agent logic, and orchestration execute locally. For enterprises in the EU dealing with GDPR requirements, or healthcare organizations under HIPAA constraints, or financial institutions managing PCI compliance, this architecture provides a viable path to AI adoption without regulatory exposure.

Production-Ready Agent Catalog

The partnership announcement highlights that Helikai's full catalog of 200-plus Helibots has been validated on RNGD hardware. This certification means enterprises can deploy these agents with confidence that performance characteristics are known and tested.

The agent catalog spans multiple domains. Legal agents analyze contracts, generate documents, and prioritize eDiscovery. Insurance agents parse policies to identify coverage requirements and gaps. IT operations agents automate invoice processing, validate purchase orders, and integrate with ERPs. Media and entertainment agents handle subtitle generation, voice dubbing, and metadata tagging.

Each agent is designed around the "one agent, one outcome" principle. An invoice processing agent extracts structured data with 99-plus percent accuracy. A contract analysis agent identifies specific clauses and flags risk factors. A semantic search agent indexes corporate knowledge bases and returns relevant results. These agents then chain together for complex workflows—invoice processing triggers address validation, credit checks, approval routing, and ERP updates in sequence.

Why This Partnership Matters

The FuriosaAI partnership validates Helikai's architectural approach and solves a practical deployment problem. During the IT Press Tour presentation, Helikai emphasized that their platform supports 40-60 different models and doesn't enforce vendor lock-in. The Furiosa certification demonstrates this model-agnostic philosophy in practice.

For developers building enterprise AI systems, this partnership offers a reference architecture that works. You get certified performance characteristics, known power consumption, validated deployment patterns, and a catalog of production-ready agents. You're not starting from scratch or hoping that your custom solution will scale.

The emphasis on power efficiency opens up deployment scenarios that weren't previously economically viable. Edge computing environments, remote facilities, and locations with limited electrical infrastructure—all become viable targets for sophisticated AI automation. The 180-watt power envelope means you can deploy meaningful AI capabilities in places where a traditional GPU server rack would be impractical.

For enterprises evaluating AI infrastructure investments, the partnership provides a concrete alternative to hyperscaler dependencies or expensive GPU buildouts. The combination of purpose-built inference hardware and domain-specific software agents creates a production-ready stack that addresses real enterprise requirements: security, predictability, efficiency, and measurable business outcomes.

The announcement includes validation from Neuralytix analyst Ben Woo, who noted that enterprises "seek to deliver the desired business benefits with a measurable ROI from their AI initiatives." This partnership between infrastructure and application layer provides exactly that—a complete stack optimized for production deployment rather than research experiments.

As AI moves from pilot projects to production workloads, partnerships like this one define what enterprise-grade actually means: certified performance, controlled environments, efficient operation, and automation that delivers business value rather than technical demos.

Integerating Webapps into Kubeflow

Sun, 01 Feb 2026 17:47:18 +0000

Kubeflow can be used not just for orchestration, kubeflow dashboard can also be used for hosting your custom webapps, or other applications.

In this case, I have installed the kubeflow as per the docs provided in the offical repository https://github.com/kubeflow/manifests.

For webapp am going to deploy this very basic application https://hub.docker.com/r/paulbouwer/hello-kubernetes/

Deploy and expose the service using these commands

kubectl run hello-app - image=paulbouwer/hello-kubernetes:1.10 - port=8080 -n kubeflow
kubectl expose pod hello-app - port=80 - target-port=8080 -n kubeflow - name=hello-service

Once the simple application is up and running, ensure the cluster IP is working by port forwarding them.

Step — 1 Once its all up and running, first we need to add our sample application in the sandwich bar of the kubeflow dashboard. Essentially the list of applications are stored in the config-map. In the centraldashboard-config, config map present in the kubeflow ns, add the following the menuLinks list

{
  "type": "item",
  "link": "/hello-app/",
  "text": "Sample App",
  "icon": "icons:stars"
}

Essentially your cm will look something like this

"menuLinks": [
        {
            "icon": "book",
            "link": "/jupyter/",
            "text": "Notebooks",
            "type": "item"
        },
        {
            "icon": "assessment",
            "link": "/tensorboards/",
            "text": "TensorBoards",
            "type": "item"
        },
        {
            "icon": "device:storage",
            "link": "/volumes/",
            "text": "Volumes",
            "type": "item"
        },
        {
            "icon": "kubeflow:katib",
            "link": "/katib/",
            "text": "Katib Experiments",
            "type": "item"
        },
        {
            "type": "item",
            "link": "/kserve-endpoints/",
            "text": "KServe Endpoints",
            "icon": "kubeflow:models"
        },
        {
              "type": "item",
              "link": "/hello-app/",
              "text": "Sample App",
              "icon": "icons:stars"
        },..
]

Once the config map is updated, ensure to rollout restart the centraldashboard deployment.

Step — 2
Create a destination roule, so they are routed to the right destination on clicking the link

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: hello-service
  namespace: kubeflow
spec:
  host: hello-service.kubeflow.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

Step — 3

Create virutal service for endpoint

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: hello-app-route
  namespace: kubeflow
spec:
  gateways:
     - kubeflow-gateway # Using your specific gateway
  hosts:
    - '*'
  http:
    - headers:
        request:
          add:
            x-forwarded-prefix: /hello-app
      match:
        - uri:
          # Using a path structure Kubeflow's auth filter often expects
            prefix: /hello-app/ 
      rewrite:
        uri: /
      route:
        - destination:
            host: hello-service.kubeflow.svc.cluster.local
            port:
              number: 80
    - match:
      - uri:
          prefix: /css/
      - uri:
          prefix: /images/
      - uri:
          prefix: /js/
      route:
      - destination:
          host: hello-service.kubeflow.svc.cluster.local
          port:
            number: 80

Ensure to add the sub url prefixes, so the things like css, images are also loaded without the 404, now even with all this you will notice that you will get a RBAC error, cos we need to add the authpolicy so kubeflow to pass the auth headers and allow the page load as iframe

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-gateway-to-sample-app
  namespace: kubeflow
spec:
  selector:
    matchLabels:
      run: hello-app  # Must match the label on your pod
  action: ALLOW
  rules:
  - from:
    - source:
        # This allows traffic specifically coming from your gateway
        namespaces: ["istio-system", "kgateway-system"]

With this you have successfully added a custom application into the kubeflow and it will look something like this

OCI Generative AI: Dedicated Cluster Sizing, Pricing, and Fine-Tuning Deep Dive

Thu, 15 Jan 2026 07:33:12 +0000

Understanding how to properly size, price, and configure dedicated AI clusters is crucial for cost-effective deployment of custom LLMs on Oracle Cloud Infrastructure. This comprehensive guide walks you through cluster types, pricing calculations, fine-tuning configurations, and how to interpret your results.

Dedicated AI Cluster Unit Types

OCI Generative AI provides several dedicated cluster unit types, each optimized for specific models and workloads:

Available Unit Types

Small Cohere Dedicated (V2)

Unit identifier: SMALL_COHERE_V2
Suitable for: Cohere Command R models
Use case: Fine-tuning and hosting smaller Cohere models

Large Cohere Dedicated

Unit identifier: LARGE_COHERE
Suitable for: Cohere Command R+ models
Use case: Larger Cohere models requiring more compute

Embed Cohere Dedicated

Unit identifier: EMBED_COHERE
Suitable for: Cohere embedding models (English V3, Multilingual V3, Light variants)
Use case: Generating embeddings at scale

Large Meta Dedicated

Unit identifier: LARGE_META (or specific like Llama2-70-count)
Suitable for: Meta Llama models (70B, 405B parameters)
Use case: Hosting and fine-tuning large Meta Llama models

Understanding Unit Requirements

Different models require different numbers of units for fine-tuning and hosting. Let's use the Cohere Command R 08-2024 model as an example:

Fine-Tuning Requirements:
To fine-tune a Cohere Command R 08-2024 model, you need a preset value of 8 Small Cohere V2 units. This is non-negotiable—the fine-tuning cluster is automatically sized for optimal training performance.

Hosting Requirements:
To host the fine-tuned Command R 08-2024 model, you need a minimum of 1 Small Cohere V2 unit. You can add more units (up to 3 or more) to handle higher call volumes.

Total for Complete Workflow:
If you want to both fine-tune and host a Command R model, you'll need:

8 Small Cohere V2 units (fine-tuning cluster)
1+ Small Cohere V2 units (hosting cluster)
Total: Minimum 9 Small Cohere V2 units

Commitment Requirements and Billing

Understanding OCI's commitment model is essential for cost planning.

Minimum Commitments

Hosting Clusters: Require a minimum commitment of 744 unit-hours per cluster

744 hours = 31 days (one month)
This is your minimum billing period for hosting

Fine-Tuning Clusters: Require a minimum of 1 unit-hour

You pay only for the actual time used
Much more flexible than hosting commitments

Billing Mechanics

Character-Based Billing (On-Demand):
A transaction is one character. 10,000 transactions = 10,000 characters. Foundational models on-demand are charged per character for prompts and responses (except embeddings, where responses aren't counted).

Unit-Hour Billing (Dedicated Clusters):
If you're hosting foundational models or fine-tuning them on dedicated AI clusters, you're charged by the unit-hour rather than by transaction.

Real-World Pricing Example: Ryan's Fine-Tuning Journey

Let's walk through a complete pricing scenario based on Bob's example from Oracle documentation, adapted for our Ryan scenario:

Ryan's Requirements

Ryan wants to:

Fine-tune a Cohere Command R 08-2024 model
Host the fine-tuned model for production use
Create a new fine-tuned version weekly (4 times per month)

Step-by-Step Cost Calculation

Step 1: Fine-Tuning Cluster Costs

Ryan creates a fine-tuning dedicated AI cluster with the preset value of 8 Small Cohere V2 units. The fine-tuning job takes 5 hours to complete. Ryan creates a fine-tuning cluster every week.

Monthly Fine-Tuning Calculation:

Fine-tuning sessions per month: 4 (weekly)
Units per session: 8 Small Cohere V2
Hours per session: 5 hours
Total unit-hours: 4 sessions × 8 units × 5 hours = 160 unit-hours

Monthly fine-tuning cost = 160 unit-hours × $<Small-Cohere-dedicated-unit-per-hour-price>

Step 2: Hosting Cluster Costs

For hosting, dedicated AI clusters require a minimum commitment of 744 unit-hours per cluster.

Let's assume Ryan uses 1 Small Cohere V2 unit for hosting:

Hosting units: 1 Small Cohere V2
Hours per month: 744 (minimum commitment)
Total unit-hours: 1 unit × 744 hours = 744 unit-hours

Monthly hosting cost = 744 unit-hours × $<Small-Cohere-dedicated-unit-per-hour-price>

Step 3: Total Monthly Cost

Total monthly cost = Fine-tuning cost + Hosting cost
Total monthly cost = (160 + 744) unit-hours × $<Small-Cohere-dedicated-unit-per-hour-price>
Total monthly cost = 904 unit-hours × $<Small-Cohere-dedicated-unit-per-hour-price>

Scaling Hosting for Higher Traffic

If Ryan decides to buy three units of Small Cohere V2 to handle higher call volume, the hosting calculation becomes:

Hosting units: 3 Small Cohere V2
Hours per month: 744 (minimum commitment applies per unit)
Total unit-hours: 3 units × 744 hours = 2,232 unit-hours

Monthly hosting cost = 2,232 unit-hours × $<Small-Cohere-dedicated-unit-per-hour-price>

Fine-Tuning Configuration Parameters

Understanding fine-tuning hyperparameters is crucial for achieving optimal model performance.

Training Methods

OCI Generative AI supports two main parameter-efficient fine-tuning methods:

T-Few (Training Few):

Parameter-efficient method for Cohere models
Selectively updates only a fraction of model weights
Recommended for small datasets (<100,000 samples)
Ideal for changing instruction-following behavior

LoRA (Low-Rank Adaptation):

Parameter-efficient method for Llama models
Adds trainable adapter matrices while freezing base model
Balances efficiency with performance
Suitable for most fine-tuning scenarios

Hyperparameters Explained

OCI Generative AI fine-tunes each base model using hyperparameters based on the pretrained base model. Start training with default hyperparameter values.

Training Epochs Total:
An epoch is one complete pass through the entire training dataset. The number of epochs determines how many times the model sees all training examples.

Too few epochs: Model underfits, doesn't learn patterns well
Too many epochs: Model overfits, memorizes training data instead of generalizing
Optimal range: Typically 3-10 epochs depending on dataset size

Learning Rate:
The learning rate influences how much model parameters are adjusted in each iteration. A lower learning rate means smaller changes to parameter estimates, requiring longer training time but potentially achieving minimum loss.

Too high: Model may overshoot optimal parameters, fail to converge
Too low: Training takes excessively long, may get stuck in local minima
Typical range: 1e-5 to 1e-3 for fine-tuning

Techniques like learning rate schedules adjust the rate over time, or adaptive algorithms like Adam or RMSprop are commonly used.

Training Batch Size:
Number of training examples processed together before updating model parameters.

Larger batches: More stable gradient estimates, faster training, higher memory usage
Smaller batches: More gradient noise (can help escape local minima), lower memory usage
Typical range: 4-32 for LLM fine-tuning

Early Stopping Patience:
Number of epochs to wait for improvement before stopping training early. Prevents wasting compute on models that have stopped improving.

Example: If patience = 3, training stops if validation loss doesn't improve for 3 consecutive epochs

Early Stopping Threshold:
The minimum improvement required to reset the patience counter. Helps distinguish real improvements from noise.

Example: Threshold = 0.001 means validation loss must improve by at least 0.001 to count as improvement

Log Model Metrics Interval Steps:
Frequency of logging training metrics (loss, accuracy) during training.

Smaller intervals: More detailed monitoring, larger log files
Larger intervals: Less granular view, smaller storage requirements
Typical value: Every 10-100 steps

Hyperparameter Calculation

The model calculates totalTrainingSteps using: totalTrainingSteps = (totalTrainingEpochs × size(trainingDataset)) / trainingBatchSize

Example Calculation:

Training dataset: 1,000 examples
Training epochs: 5
Batch size: 10

Total training steps = (5 × 1,000) / 10 = 500 steps

Understanding Fine-Tuning Results

After training completes, evaluating model performance is critical. In the model's detail page, under Model Performance, check the values for accuracy and loss. If you're not happy with the results, create another model with either a larger dataset or different hyperparameters until performance improves.

Key Metrics Explained

Accuracy:
Accuracy measures the proportion of correct predictions among the total number of cases evaluated. In the context of LLM fine-tuning, accuracy measures when the generated tokens match the annotated tokens in your training data.

Formula:

Accuracy = (Number of correct token predictions) / (Total number of tokens)

Interpretation:

Accuracy = 0.90 (90%): The model predicts the correct token 90% of the time
Higher is better, but perfect accuracy (1.0) often indicates overfitting

Example:
If fine-tuning on a medical terminology task, accuracy of 85% means the model correctly generates medical terms 85% of the time.

Loss:
Loss (also called training loss) measures how wrong the generated outputs of the model are. The loss function quantifies the discrepancy between predicted outputs and actual targets.

Key Characteristics:

Loss decreases as the model improves
Lower loss indicates better fit to training data
Should decrease consistently across epochs (if not, check learning rate)

Common Loss Functions:

Cross-entropy loss: Standard for classification and language modeling
Mean squared error: Less common for LLMs

Interpreting Loss Curves:

Training loss started at step 25 at 2.86 and by step 200, total training and validation loss was insignificant at just 0.4.

Epoch 1: Loss = 2.5
Epoch 2: Loss = 1.8  ← Good! Decreasing
Epoch 3: Loss = 1.3  ← Still improving
Epoch 4: Loss = 1.1  ← Slowing down
Epoch 5: Loss = 1.05 ← Converging

Warning Signs:

Loss increases: Learning rate too high, or data quality issues
Loss plateaus early: Learning rate too low, or model capacity insufficient
Loss fluctuates wildly: Batch size too small, or learning rate too high

Training vs. Validation Loss

Training Loss:
Measured on the data the model is learning from. Always decreases with enough training.

Validation Loss:
Measured on held-out data the model hasn't seen. Indicates generalization ability.

Ideal Pattern:

Epoch  | Training Loss | Validation Loss
-------|---------------|----------------
1      | 2.5          | 2.6
2      | 1.8          | 1.9
3      | 1.3          | 1.4
4      | 1.1          | 1.2

Both decreasing = healthy training

Overfitting Pattern:

Epoch  | Training Loss | Validation Loss
-------|---------------|----------------
1      | 2.5          | 2.6
2      | 1.8          | 1.9
3      | 1.3          | 1.6  ← Warning!
4      | 1.0          | 1.8  ← Overfitting!

Training loss decreases but validation loss increases = model memorizing training data

Additional Evaluation Metrics

While accuracy and loss are primary, consider these secondary metrics:

Perplexity:
Exponential of the loss, more interpretable for language models. Lower is better.

Perplexity = e^(loss)
If loss = 1.5, perplexity = 4.48

F1-Score:
The F1-score is a combined measure of recall and precision, offering a balanced assessment of LLM performance in language tasks.

Particularly useful for classification tasks within fine-tuning (e.g., intent classification, sentiment analysis).

BLEU/ROUGE Scores:
For text generation quality, though traditional metrics like BLEU and ROUGE are considered insufficient for natural conversations.

Best Practices for Cost Optimization

1. Fine-Tuning Strategy

Minimize Training Sessions:

Prepare high-quality datasets before fine-tuning
Test hyperparameters on small subsets first
Batch multiple improvements into single training runs

Optimize Training Time:

Use early stopping to avoid unnecessary epochs
Monitor loss curves—stop if converging
Start with smaller epochs, increase if needed

2. Hosting Strategy

Right-Size Hosting Units:
Start with 1 unit and add more only as call volume demands. Remember the 744-hour minimum commitment.

Consider On-Demand for Low Volume:
If usage is sporadic or low, on-demand pricing might be more cost-effective than dedicated hosting.

Calculation for Break-Even:

On-demand cost per 10k characters: $X
Dedicated cost per unit-hour: $Y
Characters processed per hour: Z

Break-even: Z × $X = $Y

3. Development vs. Production

Development/Testing:

Use on-demand inferencing
Fine-tune infrequently with small datasets
No hosting commitment needed

Production:

Dedicated hosting for consistent performance
Batch fine-tuning updates (weekly/monthly)
Scale hosting units based on actual traffic

4. Monitoring and Optimization

Track Key Metrics:

Cost per fine-tuning session
Cost per inference (characters processed)
Utilization rate of hosting clusters
Model performance metrics (accuracy, loss)

Optimization Opportunities:

Reduce fine-tuning frequency if model performance stable
Adjust hosting capacity based on traffic patterns
Use cached responses for common queries

Common Pitfalls and Solutions

Pitfall 1: Over-Fine-Tuning

Problem: Creating new fine-tuned models too frequently
Cost Impact: Unnecessary compute charges
Solution: Fine-tune only when you have substantial new training data or performance degrades

Pitfall 2: Over-Provisioning Hosting

Problem: Allocating too many hosting units "just in case"
Cost Impact: Paying for unused capacity (744 hours/month minimum per unit)
Solution: Start with minimum units, monitor utilization, scale up as needed

Pitfall 3: Ignoring Model Metrics

Problem: Not evaluating accuracy and loss before deploying
Cost Impact: Poor model performance requiring additional fine-tuning sessions
Solution: Always check Model Performance metrics and iterate if needed

Pitfall 4: Poor Hyperparameter Choices

Problem: Using default hyperparameters without experimentation
Cost Impact: Suboptimal models requiring more training sessions
Solution: Test hyperparameters systematically, document what works

Practical Workflow Example

Here's a complete workflow with cost awareness:

Phase 1: Initial Fine-Tuning (Week 1)

1. Prepare 5,000 training examples
2. Create fine-tuning cluster (8 Small Cohere V2 units)
3. Fine-tune with default hyperparameters (5 hours)
4. Cost: 40 unit-hours × $price
5. Check accuracy: 82% ← Needs improvement

Phase 2: Iteration (Week 2)

1. Increase dataset to 10,000 examples
2. Adjust learning rate from 1e-4 to 5e-5
3. Fine-tune again (6 hours)
4. Cost: 48 unit-hours × $price
5. Check accuracy: 91% ← Acceptable!

Phase 3: Deployment (Week 3-4)

1. Create hosting cluster (1 Small Cohere V2 unit)
2. Deploy fine-tuned model
3. Cost: 744 unit-hours × $price (monthly minimum)
4. Monitor usage and performance

Total Month 1 Cost:

Fine-tuning: (40 + 48) = 88 unit-hours
Hosting: 744 unit-hours (minimum commitment)
Total: 832 unit-hours × $<price per unit-hour>

Conclusion

Effective use of OCI Generative AI dedicated clusters requires understanding three key areas:

1. Sizing: Match cluster unit types and counts to your specific models

Fine-tuning: 8 Small Cohere V2 units for Command R
Hosting: Start with 1 unit, scale based on traffic

2. Pricing: Plan for commitments and optimize costs

Fine-tuning: Pay for actual hours used (minimum 1 hour)
Hosting: 744-hour monthly minimum per cluster
Calculate break-even vs. on-demand based on usage

3. Configuration: Tune hyperparameters and monitor metrics

Start with defaults, iterate systematically
Watch accuracy (higher is better) and loss (lower is better)
Use early stopping to save compute time

By following these guidelines, you can deploy custom fine-tuned models on OCI cost-effectively while maintaining high performance.

Building an intelligent data fabric that actually works: CTERA's pragmatic approach to enterprise AI

Wed, 08 Oct 2025 18:59:02 +0000

Building an Intelligent Data Fabric That Actually Works: CTERA's Pragmatic Approach to Enterprise AI

Why 95% of Enterprise AI Projects Fail (And How to Fix It)

CTERA presented at the 64th IT Press Tour with a simple message: most enterprise AI initiatives crash because of bad data, not bad models. After spending 17 years building distributed file systems for Fortune 500 companies and government agencies, they've seen this pattern repeat.

The problem isn't ChatGPT. It's the mess of PDFs, Excel files, and Word documents scattered across your infrastructure.

The Architecture: Three Layers That Make Sense

CTERA's approach breaks down into three technical stages:

Wave 1: Global Namespace

Instead of fighting data silos, they built a software-defined global namespace over object storage. Think of it as a unified API layer that works with file (CIFS/NFS) and object (S3) protocols. The smart part: edge filers cache frequently accessed data locally, while storing everything in cheap object storage. Your users get local performance. Your CFO gets object storage pricing.

The system uses a hub-and-spoke model with real-time synchronization. When data changes at any edge location, a notification service publishes updates via a pub-sub API. This matters because AI training pipelines need fresh data, not stale snapshots.

Wave 2: Metadata Intelligence

Here's where it gets practical. CTERA added real-time monitoring at the block and file level. Their Ransom Protect feature uses AI to detect anomalies in file access patterns - things like unusual encryption activity or mass file modifications. When it spots something suspicious, it can automatically cut off access and roll back to immutable snapshots.

Their Insight product gives you a SaaS dashboard showing audit logs, access patterns, and forensics going back a year. It's built on AWS with multi-tenant isolation.

The clever bit: They added Model Context Protocol (MCP) support in June 2025. This means you can connect Claude, ChatGPT, or any MCP-compatible client directly to your file system. Ask "What contracts mention petroleum?" and it searches your data, respecting existing ACLs.

Wave 3: Intelligent Data Fabric

This is the AI piece that actually works in production. Instead of vectorizing everything and hoping for the best, they built a data curation pipeline:

Timely Ingestion: Collectors at edge sites capture data from NFS, SMB, and S3 sources as it's created
Format Unification: Convert everything to markdown - PDFs, Office docs, even transcribed audio and OCR'd images
Metadata Enrichment: Use vision models to extract structured fields from unstructured documents
Data Filtering: Drop files with PII, confidential stamps, or other risky content based on your guardrails
Vectorization: Only after cleaning, index into your vector database

The result: curated datasets that won't poison your RAG implementations.

Real Implementation: Medical Law Firm Case

One customer analyzes medical malpractice cases. Previously, they paid doctors $1,000+ to manually review hundreds of documents per case.

Now they:

Use vision models to analyze scanned medical records (including handwritten notes)
Extract structured metadata: doctor names, exam dates, findings, procedures
Store everything in a searchable schema
Generate comprehensive encounter reports automatically

The system cut their per-case analysis costs from thousands to hundreds of dollars.

The MCP Integration Details

CTERA built both MCP client and server capabilities:

As MCP Client: Their "experts" (virtual employees) can invoke any MCP tool. Send emails, query databases, search the web, generate images. Think of it as giving your AI agents hands.

As MCP Server: External tools can invoke CTERA experts. You can use Claude Desktop, Cursor, n8n - anything MCP-compatible - to search your enterprise data. The key: it respects your existing file permissions. No shadow copies. No surprise data leaks.

What They Got Right

The notification service architecture is smart. By keeping CTERA out of the data path for reads, they avoid becoming a bottleneck. Your AI training jobs can read directly from object storage once they have the metadata.

The permission-aware design matters for regulated industries. When your AI assistant queries data, it only sees what that user has access to. Banks and healthcare providers actually care about this.

The staged approach makes sense. You don't need to buy everything. Start with the global namespace for cost savings. Add security features when you're ready. Layer on AI capabilities when you have actual use cases.

What to Watch

This is still early. Their Data Intelligence product is new. The real test: Can business users actually create and maintain their own "experts" without developer help? The demos look good, but production is different.

The other question: How do you evaluate quality? They talk about having customers provide sample questions and correct answers, then tuning the system to match. That's the right approach, but it requires mature ML ops practices most enterprises don't have yet.

The Bottom Line

CTERA's been building enterprise storage for 17 years. They understand distributed systems, security, and what actually ships. Their AI approach feels pragmatic - fix the data quality problem first, then worry about fancier models.

If you're trying to get GenAI working on real enterprise data, the architecture patterns here are worth studying. Especially the data curation pipeline and permission-aware access.

Your CTO can't tell you how much AI code you're writing—and that's a bigger problem than you think.

Tue, 16 Sep 2025 19:41:42 +0000

Why Your CTO Can't Tell You How Much AI Code You're Actually Writing

Span's new AI code detector reveals the measurement gap that's keeping engineering leaders in the dark about their AI transformation.

Engineering leaders are flying blind. They're making million-dollar decisions about AI coding tools based on guesswork, self-reported surveys, and vendor claims that don't match reality.

This disconnect became clear when I spoke with J Zac Stein and Henry Liu, co-founders of Span, about their new AI code detector. The tool can identify AI-assisted versus human-written code with over 95% accuracy—and it's revealing some uncomfortable truths about how we measure AI adoption.

The Numbers Don't Add Up

Google claims 25% of new code comes from AI. Microsoft says 30%. But when engineering teams try to verify these numbers internally, they hit a wall. Current measurement methods count lines of code suggested by AI tools, but they don't track what happens next. Was that code modified before commit? Deleted entirely? No one knows.

"Engineering leaders are facing immense pressure to demonstrate the value of AI investments, but they're making decisions based on anecdotal evidence," Henry Liu told me. This pressure comes from boards and executives who want concrete ROI metrics, not developer surveys.

The problem runs deeper than bad data. Companies are operating over a dozen different AI coding tools simultaneously—Copilot, Cursor, Claude, and more. Each tool provides different telemetry. Some provide none at all. And none of them can track the significant amount of AI-generated code that comes from developers copying and pasting from ChatGPT.

Beyond the Hype

The biggest misconception executives have about AI coding? That faster code writing automatically equals higher productivity. This assumption ignores critical factors like code review burden, technical debt, and defects in production.

"Writing code is not the only thing engineers do," Liu explained. "The longer the company, the less writing code is a bottleneck." Senior engineers spend more time on architecture decisions, coordination, and ensuring teams work on the right problems. A 30% speed increase in coding doesn't translate to 30% overall productivity gains.

This reality creates tension between believers and skeptics within engineering teams. Leadership pushes top-down AI adoption while senior individual contributors remain cautious. Without solid data, these debates rely on anecdotal evidence and personal preferences.

How the Detection Works

Span's solution centers on span-detect-1, a machine learning model trained on millions of code samples with known provenance. The model analyzes patterns in token sequences, syntax quirks, and stylistic regularities to determine AI versus human authorship.

The technical challenge was significant. When reasoning models like ChatGPT improved, the team had to dramatically increase their training data—by 10x—to maintain accuracy. Many research papers in this space don't properly evaluate their models, leading to inflated accuracy claims that don't hold up in real-world conditions.

Currently supporting Python, TypeScript, and JavaScript, the detector works across all AI coding tools. This tool-agnostic approach helps companies understand their true AI usage regardless of which specific tools their developers prefer.

What Companies Are Learning

Early customers are discovering adoption patterns that challenge assumptions. While senior engineers tend to be more reserved about AI tools, the differences aren't as stark as expected. Teams working on greenfield projects adopt AI more readily than those maintaining legacy codebases, but overall adoption is climbing across all segments.

For companies that have fully embraced AI coding tools, weekly usage rates average around 70%. Some engineers are spending hundreds of dollars daily on token costs. As Liu noted, "Token spend is going to be comparable to payroll spend in the future."

This shift transforms every engineer into a resource allocator. Understanding which AI tools provide the best ROI becomes critical for both productivity and cost management—similar to how companies had to learn cloud cost optimization.

The Bigger Picture

Span's AI detector represents just one piece of a broader developer intelligence platform. The company is building what Liu and Stein call "the tool we wish we'd had in our previous roles"—a comprehensive suite for understanding engineering productivity in the AI era.

Their approach goes beyond simple metrics. Span uses AI to understand developer productivity rather than relying on outdated measures like lines of code. The platform includes qualitative surveys to identify developer headwinds and tailwinds, cost capitalization tools, and upcoming features to compare defect rates between AI and human code.

"This type of tool was not possible three years ago," Liu said. "Span helps you lead your engineering organization through AI transformation with data, not hype."

As AI coding tools continue evolving rapidly, the need for objective measurement becomes more critical. Companies can't optimize what they can't measure. Span's detector provides the ground truth that engineering leaders need to make informed decisions about their AI strategy.

The future of software development is clearly AI-augmented. The question isn't whether to adopt these tools—it's how to do it intelligently, with data instead of hope.

Only 2% of companies are ready to scale AI securely—here's what the other 98% are missing.

Mon, 14 Jul 2025 11:00:34 +0000

The AI Readiness Reality Check: Why 98% of Organizations Aren't Ready to Scale

New research from F5 Networks delivers a sobering reality check for the enterprise AI landscape: while 96% of organizations are deploying AI models, only 2% qualify as "highly ready" to scale their AI initiatives securely. The findings, based on surveys of 650 global IT leaders and 150 AI strategists, reveal a massive gap between AI adoption and operational readiness that has significant implications for development teams.

The State of AI Infrastructure Today

According to Lori MacVittie, F5's Distinguished Engineer who led the research, the problem isn't lack of adoption—it's the fragmented approach organizations are taking. "The average organization uses three AI models, and this correlates with multi-environment deployments," MacVittie explains. "But most haven't solved the complexity and inconsistencies that come with that."

The data shows 25% of applications on average now use AI, but there's a stark divide in how organizations approach implementation. Highly ready organizations demonstrate "portfolio-wide saturation" while low-readiness organizations typically use AI in fewer than 25% of their applications, usually in isolated or experimental settings.

What sets the 2% of highly ready organizations apart? MacVittie points to several key factors: superior data practices, experience with diverse application types, and mature infrastructure management. "They understand how to formally manage data, how to label it correctly, and then secure it," she notes. "Larger organizations that have been doing this forever understand how to label data and use it and mine it already. So AI is just another thing they're going to mine, label, and secure."

The Security Paradox

Perhaps the most striking finding is a security paradox: 71% of organizations use AI to boost their security operations, but only 31% deploy AI-specific cybersecurity protections. This disconnect creates significant risks for development teams.

"Most security solutions have always had some form of AI in them already," MacVittie explains. "But the lack of AI-specific protections is concerning. You've got to defend against AI attackers, and they're not putting the types of AI protections that they need to identify, to watch, to do deeper inspection, and then be able to actually stop it."

The challenge is that AI attackers behave differently than traditional bots. "A normal bot will just do the URI as it can see, it follows expected paths. But an AI can take different paths. It can actually read the page and make a different choice. How do you distinguish between me, who's fumbling around on the website, and that AI?"

Data Labeling: The Foundation Missing

Only 24% of organizations practice formal data labeling—a critical gap that MacVittie identifies as essential for both transparency and preventing adversarial attacks. For engineering teams just starting to implement this, she recommends beginning with operational data labeling.

"Simple things like: this is an AI request from software bot, AI from human, part of a bigger process," MacVittie suggests. "Which process is this part of, the order process or the query process? Because these are two different security policies, I have to apply."

The research found that organizations taking a continuous approach to data labeling significantly outperform those using ad-hoc methods. "This kind of data labels are going to help security so much if you get it nailed down early," MacVittie emphasizes.

Cross-Cloud Complexity

As organizations deploy AI across multiple environments, governance gaps emerge. MacVittie provides a concrete example: "When you've got models in different places—maybe you're using OpenAI as a service and on-premise you might have an open source model like Mistral—you've got two different locations, two different models, and you've got two different security policies. So now you have inconsistency in security."

This fragmentation extends beyond just security policies to include access controls, data handling procedures, and basic AI governance—creating a complex web of inconsistencies that development teams must navigate.

The Observability Gap

One of the most critical missing pieces in AI infrastructure is what MacVittie calls "semantic observability." Current monitoring approaches are inadequate for AI systems.

"We monitor for up, down, fast, slow, somebody attacked, somebody kept making this request. But nowhere today do we really log things like what was the request. The prompt is now the context that we need to understand," she explains. "We need to be able to see that level and deeper—how did it answer and why."

This represents a fundamental shift for infrastructure teams. Traditional observability focused on system metrics, but AI requires understanding the semantic content of interactions between systems.

Technical Debt and Fragmentation

For the 77% of organizations classified as "moderately ready," the primary barrier to advancement is technical debt and infrastructure fragmentation. "Their infrastructure is fragmented," MacVittie notes. "We've got 35 different delivery and security things. We've got five WAFs. We've got three different kinds of load balancers. In order to scale, enterprises need standardization."

This fragmentation becomes particularly problematic as organizations try to implement consistent AI governance across diverse toolsets and environments.

Rethinking Application Architecture

MacVittie argues for a fundamental shift in how development teams think about AI systems. "We need to really embrace that these are all applications. Every single thing is an application. You're doing application to application," she explains.

This philosophical change has practical implications. Traditional client-server models break down in AI environments, where context flows bidirectionally and systems can switch roles dynamically. "These connections are going to go all over the place, so we've got to come to an understanding that's not how communication works anymore."

The Context Challenge

Perhaps the most significant architectural challenge developers face is managing context as state. "State is not stateless in AI systems. They carry the state with them," MacVittie explains. "As you're building an application, you have to be aware that as you're exchanging messages with other systems, especially other AI, that context is your state, and it might be corrupted. It might get too big."

This represents a fundamental shift from traditional stateless web architectures. Context in AI systems isn't just key-value pairs—it's large, complex text that must be actively managed throughout the application lifecycle.

The Path Forward

For development teams looking to improve their AI readiness, MacVittie recommends starting with the fundamentals: implement continuous data labeling processes, standardize infrastructure where possible, and begin building semantic observability capabilities.

"If you don't have a good observability practice right now, and most people don't because it was an afterthought, you need to lay it down now so that you're ready to be able to do this deeper logging and monitoring of what's going on as all this AI is talking to each other."

The window for addressing these challenges is narrowing. As AI adoption accelerates and agentic systems become more common, organizations that haven't built proper foundations will find themselves increasingly unable to compete. The 2% who are highly ready today aren't just ahead in deployment—they're building the infrastructure that will define the next generation of AI-powered applications.

Cloud-aware platform helps achieve 92% GPU utilization while slashing infrastructure costs for AI workloads.

Sat, 10 May 2025 12:01:54 +0000

Volumez: Reinventing Cloud Infrastructure for AI/ML Workloads

The increasing complexity of AI/ML workloads is exposing critical inefficiencies in how cloud infrastructure is provisioned and managed. A Silicon Valley startup called Volumez is tackling this problem with a novel approach that could revolutionize how engineers architect their AI infrastructure stacks.

The Infrastructure Imbalance Problem

Traditional cloud infrastructure for AI workloads typically suffers from several key inefficiencies:

I/O bottlenecks: GPUs often sit idle waiting for data
Overprovisioning: Engineers compensate for bottlenecks by purchasing excess capacity
Configuration complexity: Optimal cloud configuration is beyond human capability
Operational overhead: Data scientists get dragged into infrastructure management
Cost inefficiency: Cloud bills skyrocket without corresponding performance gains

According to Volumez's internal benchmarks, these inefficiencies can lead to GPU utilization rates below 80%, effectively doubling the cost of model training while extending development timelines.

A New Architecture: Data Infrastructure as a Service

Volumez has developed what they call "Data Infrastructure as a Service" (DIaaS), which fundamentally rethinks how cloud resources are configured for AI workloads.

The key innovation is what they call "cloud awareness" - deep profiling of cloud provider capabilities to create balanced infrastructure that precisely matches workload requirements without proprietary components in the data path.

# Traditional approach - manual configuration
# Configure storage volumes
aws ec2 create-volume --volume-type io2 --iops 50000 --size 4000 --region us-west-2
# Configure network settings
aws ec2 modify-instance-attribute --instance-id i-1234567890abcdef0 --ebs-optimized

# Volumez approach - declarative configuration
import volumez as vlz

# Define policy
policy = {
  "name": "training-infrastructure",
  "performance": {
    "iops": 1000000,
    "latency_usec": 300
  },
  "encryption": True,
  "resilience": {
    "zones": 1
  }
}

# Create infrastructure
vlz.create_infrastructure(policy)

Technical Implementation Details

The implementation relies on three key components:

1. Cloud Profiling Engine: Conducts deep analysis of cloud provider components, including:

Physical server topology
Network fabric characteristics
Storage media specifications
Cost structures

2. Configurator Agent: A lightweight user-space process that:

Implements optimal Linux configurations
Adjusts queue depths
Configures multipathing
Sets up resilience schemes

3. Declarative API Layer: Enables infrastructure definition through:

PyTorch library extensions
Terraform/Bicep modules
REST APIs
Kubernetes operators

The critical insight is that by precisely understanding cloud components and their interactions, a SaaS system can automatically create optimal Linux configurations without introducing proprietary storage controllers or other middleware.

Performance at Scale

In MLCommons MLPerf Storage 1.0 benchmarks, Volumez demonstrated:

1.14 TB/sec throughput
9.9M IOPS
92% GPU utilization
411 simulated GPUs

These results significantly outperformed traditional storage approaches from vendors like DDN, NetApp, and VAST Data, while showing cost reductions of 27-70% for storage and 50-92% for compute compared to standard AWS configurations.

Implementation Patterns

The platform offers two main deployment patterns that engineers should consider:

1. Hyperconverged Pattern
┌─────────────────┐ ┌─────────────────┐
│ GPU Instance │ │ GPU Instance │
│ ┌─────┐ ┌─────┐ │ │ ┌─────┐ ┌─────┐ │
│ │ GPU │ │ GPU │ │ │ │ GPU │ │ GPU │ │
│ └─────┘ └─────┘ │ │ └─────┘ └─────┘ │
│ │ │ │ │ │ │ │
│ ┌─────────────┐│ │ ┌─────────────┐│
│ │ Local SSDs ││ │ │ Local SSDs ││
│ └─────────────┘│ │ └─────────────┘│
└─────────────────┘ └─────────────────┘

Uses local SSDs on GPU servers
Best for datasets under 100TB
Simpler deployment model
Good for persistent clusters

2. Flex Pattern
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ GPU Cluster │ │ GPU Cluster │ │ GPU Cluster │
│ ┌─────────┐ │ │ ┌─────────┐ │ │ ┌─────────┐ │
│ │ GPU GPU │ │ │ │ GPU GPU │ │ │ │ GPU GPU │ │
│ └─────────┘ │ │ └─────────┘ │ │ └─────────┘ │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘

   │                │                │
   │                │                │
   └────────────────┼────────────────┘
                    │
                ┌───▼───┐

┌──────────────┐ │Network│ ┌──────────────┐
│ Storage Node │◄───┤ ├───►│ Storage Node │
│ ┌──────────┐ │ └───────┘ │ ┌──────────┐ │
│ │ SSD SSD │ │ │ │ SSD SSD │ │
│ │ SSD SSD │ │ │ │ SSD SSD │ │
│ └──────────┘ │ │ └──────────┘ │
└──────────────┘ └──────────────┘

Scales storage independently from compute
Supports datasets >100TB
Better for dynamic clusters
Higher resilience options

Developer Experience

For developers and data scientists, the most compelling feature is the ability to provision infrastructure directly from Python notebooks:

import volumez as vlz
from torch.utils.data import DataLoader

# Create managed dataset with automated infrastructure
dataset = vlz.datasets.ImageFolder(
    dataset_name="medical-scans",
    mode=vlz.datasets.Mode.Train,
    version=vlz.datasets.Version.Latest,
    credentials=credentials,
    mount="/mnt/dataset"
)

# Infrastructure automatically provisioned and optimized
dataloader = DataLoader(dataset, batch_size=32, shuffle=True)

# Process data normally
for epoch in range(10):
    for i, batch in enumerate(dataloader):
        # train step
        pass

# Infrastructure automatically torn down when complete
dataset.close()

This eliminates the need for separate MLOps or infrastructure teams, allowing data scientists to work independently while maintaining optimal performance.

Engineering Considerations

When evaluating this approach for your own AI infrastructure, consider:

1. Workload characteristics

File sizes and counts
Data access patterns
Batch sizes and preprocessing needs

2. Team structure

Data scientist to MLOps engineer ratio
Team experience with infrastructure
Preferred workflow tools

3. Cost structure

GPU utilization targets
Infrastructure budget constraints
Performance requirements

Conclusion

Volumez's approach represents a significant rethinking of how cloud infrastructure should be configured for AI workloads. By leveraging deep cloud awareness and standard Linux components, their platform offers a compelling alternative to traditional storage-centric approaches.

For engineering teams struggling with AI infrastructure bottlenecks, particularly those dealing with large-scale training workloads, this cloud-aware approach could offer substantial performance improvements while reducing both infrastructure costs and operational complexity.

The most impressive aspect may be how Volumez manages to achieve these outcomes without introducing proprietary middleware or storage controllers - instead, they're simply making better use of what the cloud already provides through intelligent configuration.

Fabrix.ai automates IT operations through AI agents that reason, decide and act—solving complex operational challenges.

Wed, 07 May 2025 16:03:13 +0000

Fabrix.ai Revolutionizes IT Operations with Agentic AI Framework

The acceleration of digital transformation has created increasingly complex IT environments that human operators struggle to manage effectively. As systems generate terabytes of data daily, traditional monitoring tools and even basic AI solutions fall short of addressing the challenges of modern IT operations. Fabrix.ai (formerly CloudFabrix) has introduced a groundbreaking approach to this problem with their Modern Operational Intelligence Platform powered by Agentic AI.

Unlike conventional generative AI that simply responds to prompts, agentic AI empowers autonomous agents to reason about problems, make decisions, and take corrective actions with minimal human intervention. This innovation represents a significant leap forward for developers, engineers, and architects seeking to automate complex IT operations.

The Three Pillars of Fabrix.ai's Platform

Fabrix.ai's platform integrates three essential components that work synergistically to enable autonomous operations:

Data Fabric

The foundation of the platform is the Robotic Data Automation Fabric (RDAF), which provides:

Integration with over 1,000 data sources through built-in "bots"
Data ingestion, transformation, and enrichment capabilities
Telemetry pipelines for routing data to various destinations
Support for structured, unstructured, and real-time data

This layer ensures that AI agents have access to comprehensive, context-rich data from across the IT environment, whether it resides in on-premises systems, cloud services, or specialized tools.

Automation Fabric

The middle layer provides an outcome-driven workflow framework that:

Orchestrates workflow execution by agents
Supports both fully autonomous operations and human-in-the-loop scenarios
Offers dynamic and extensible workflow templates
Integrates with third-party automation tools like Ansible, Terraform, and Cisco BPA

This fabric enables agents to execute complex sequences of actions across different systems and platforms without requiring manual intervention.

AI Fabric

The heart of the platform is the AI Fabric, which:

Enables building and deploying AI agents using large language models (LLMs)
Provides agent orchestration and lifecycle management
Enforces guardrails and quality controls to ensure agents operate safely
Supports causal reasoning and context-aware analysis

// Conceptual example of creating an agent with Fabrix.ai
const eventMonitorAgent = {
task: "Monitor all change requests for ACL changes to networking infrastructure.

     If a specific ACL change is blocking access to critical endpoints or 
     opening access to risky assets, update the change request to block the change.",

schedule: "Every 60 minutes",
dataAccess: ["network_devices_inferred", "acl_assignments", "network_interfaces"],
automation: ["update_ticket", "notify_team"]
};

How Agentic AI Differs from Traditional Approaches

Traditional IT operations rely on rule-based systems or simple machine learning models that identify anomalies but require human interpretation and action. Even advanced AIOps platforms typically generate alerts or recommendations rather than taking autonomous action.

Fabrix.ai's agentic approach fundamentally changes this paradigm:

Traditional ML Approach

Identifies anomalies based on statistical patterns
Requires substantial training data
Focuses on detection, not resolution
Limited ability to explain findings
Requires continuous retraining

Agentic AI Approach

Detects anomalies with contextual understanding
Can operate with less historical data
Autonomously recommends or executes corrective actions
Offers greater explainability of decisions
Adapts and learns from each execution

Real-World Use Cases

The platform supports several critical IT operations scenarios:

1. Anomaly Detection and SLO Management

Agents monitor network traffic and system metrics, alerting on unusual patterns that might indicate security breaches or performance issues. Unlike traditional monitoring tools, these agents can understand the business impact of anomalies and take appropriate action.

2. Network Digital Twin

Agents create virtual replicas of network infrastructure to:

Establish baselines for normal operation
Run what-if scenarios for planned changes
Generate predictive maintenance schedules
Validate access control list (ACL) changes

3. Closed-Loop Remediation

When problems occur, agents can:

Automatically detect failed applications or infrastructure issues
Provision or scale resources to meet increased demand
Implement configuration changes to resolve problems
Document actions taken and update relevant tickets

Note: Organizations can create customized agents using conversational prompts without extensive coding, making advanced automation accessible to teams without specialized AI expertise.

Implementation Architecture

Fabrix.ai's platform can be deployed in various configurations:

On-premises deployment leveraging local GPU resources
Cloud-based deployment in the customer's VPC
Hybrid approach with components in both environments

The platform integrates with NVIDIA's inference microservices (NIM) for efficient model execution and supports multiple LLM backends, including open-source models and commercial offerings.

Real-World Impact

Early adopters report significant benefits:

90%+ reduction in alert noise
Streamlined operations across organizational silos
Improved incident response times
Enhanced visibility for both operations teams and executives
CMDB accuracy improvements through continuous validation

A service provider managing 1,900 global customers implemented Fabrix.ai and transformed their operations from fragmented, tool-heavy workflows to a unified approach with real-time visibility and automated remediation.

The Development Experience

For developers and engineers, Fabrix.ai provides:

A visual IDE for agent creation and testing
The ability to inspect and modify generated task graphs
Visibility into agent reasoning and decision points
Integration with existing automation frameworks
Comprehensive storyboards for monitoring agent performance

// Example of agent task graph structure
{
"agent": {

"name": "acl_agent",
"tasks": [
  {
    "id": "task_1",
    "type": "stream_query",
    "stream": "acls, network_devices_inferred, acl_assignments",
    "filters": {
      "acl_id": 110,
      "device_type": ["Laptop", "Workstation", "Server"],
      "device_hostname": "Router01"
    }
  },
  {
    "id": "task_2",
    "type": "llm_generator",
    "depends_on": ["task_1"],
    "prompt": "Analyze ACL rules for security risks..."
  },
  // Additional tasks in the workflow
]

}
}

Conclusion

Fabrix.ai represents a significant advancement in IT operations automation by combining data integration, workflow automation, and agentic AI into a cohesive platform. For engineers and architects struggling with the increasing complexity of modern IT environments, this approach offers a path toward truly autonomous operations. While traditional AIOps tools have helped with detection and analysis, Fabrix.ai's agentic framework takes the next logical step by enabling AI systems to reason about problems and take appropriate action. As organizations continue to face talent shortages and growing operational complexity, autonomous agents may become an essential part of IT operations strategy.