Coder Legion - Recent posts tagged datamanagement

$360,000 a Petabyte and 105 Days to Get Out: Microsoft's New Math for University Storage

Tue, 25 Aug 2026 12:00:00 +0000

For years, Microsoft 365 storage for universities was unlimited and free. That changed this year. Microsoft capped Education tenants at 100 terabytes plus 50 to 100 gigabytes per licensed user, and turned on overage pricing worth roughly $360,000 per petabyte per year once that quota is reached. Universities holding two or three petabytes of data, which is common at large research institutions, are looking at bills in the hundreds of thousands to over a million dollars a year for storage they never budgeted for.

"Shock. Horror. Panic." That's how Jason Lohrey, CEO and founder of data management company Arcitecta, describes the week an IT department first sees that number. "They are powerless and wedged unless they can get their data out in time," he says. Some are considering deleting data in bulk just to avoid the cost, a move he calls risky given how much of that data is research output, intellectual property, or records under retention rules.

The math gets worse before it gets better. Microsoft also throttles how fast data can leave a tenant, at 400 gigabytes an hour, tenant-wide, across every backup tool and application token combined. At that rate, moving one petabyte out takes roughly 105 days. Universities holding several petabytes are looking at the better part of a year, and overage pricing starts hitting renewals from December 2026 onward. For institutions just now realizing the size of the problem, Lohrey is blunt: "it's likely too late for many to escape unscathed." A second clock is running underneath that one. Microsoft moves accounts inactive for 12 months to read-only, archives them soon after, and permanently deletes them starting mid-2027, putting former students' and staff's research data at risk right along with them.

This is the backdrop for Mediaflux Connect 365, Arcitecta's new platform for managing OneDrive, SharePoint, and Teams data, announced August 25. The announcement itself is less interesting than what it implies operationally, and why the same cost pressure now forcing this issue also happens to be pushing university data toward being usable for AI.

Lohrey draws a direct comparison to self-managed storage gone wrong: data goes in, nobody looks at it again, and years later nobody remembers what's there or why they're still paying for it. "Active data management" is meant to be the opposite: a system that keeps evaluating data against a goal, like cost, and acts on it automatically. Data past its useful life gets flagged for deletion. Data nobody has touched in months moves to cheaper storage on its own. "Active systems will use business rules and AI to ensure the value embodied in the data is realised," Lohrey says.

Doing that without breaking things for end users starts with a scan cross-referenced against Active Directory or LDAP, to find accounts belonging to people who've already left the university. Those move first, since nobody's coming back Monday morning to look for that data. Everything that moves stays reachable: if a drive gets relocated and someone still needs it, the data isn't gone, it's just sitting in Mediaflux instead of OneDrive.

The 400GB-an-hour cap doesn't get solved by moving data faster. It gets solved by moving less of it. Lohrey describes prioritizing what's actually worth the bandwidth: skip duplicate copies, skip anything that's easily regenerated, move what matters most first. "It's easier to be intelligent when you understand what you are dealing with," he says, which is why the analysis runs before anything else moves.

That analysis step is also where the AI story shows up, and Lohrey is candid that cost, not AI ambition, is what actually got universities to pay attention. "In reality, it's cost alone that appears to have been the catalyst," he says. "Embracing the opportunity, we have introduced the perfect triple: cost reduction, data management, and AI." In practice, the same metadata used to decide where a file should live also gets indexed as searchable vectors inside XODB, Arcitecta's own database engine, built specifically to handle billions of files at petabyte scale. A dataset that gets pulled off OneDrive to cut costs ends up more searchable, and more usable for AI workloads, than it ever was sitting in a SharePoint folder.

None of this is unique to Microsoft, and Lohrey doesn't pretend otherwise. Google made a similar move on free storage years ago. "Anything that looks 'free' is not free," he says. "It's a hook to get you locked in." Arcitecta says the reactions it's hearing from prospects range from resigned to something closer to relief. One university, Lohrey says, told them this was "a chance to get out of Microsoft" entirely.

For IT teams watching this play out at their own institution, the real lesson isn't about Microsoft 365 licensing specifically. It's a prompt to ask a more basic question about every platform holding institutional data: who actually controls it, and what happens the next time the terms change.

From Storage Vendor to Data Partner: A Conversation with Everpure's Nirav Sheth

Thu, 18 Jun 2026 21:30:43 +0000

The first thing a company typically discovers when it deploys Everpure Data Intelligence isn't what it expected. It's data it didn't know it had.

Nirav Sheth, VP of Worldwide Systems Engineering at Everpure, described a proof of concept that illustrates the pattern well. A customer connected Data Intelligence to their environment and found sensitive financial data sitting in a Google Drive folder with open permissions — visible to anyone in the company, with no governance controls of any kind. Nobody had flagged it. Nobody had even known it was there.

"I ask customers: do you have a really good handle of all your data across all your applications from the edge to the data center to the cloud?" Sheth told me at Pure Accelerate 2026. "And many actually just start to laugh as an acknowledgment of, yeah, you got me."

That moment — when a customer realizes the scope of what they don't know — is where Sheth's work actually begins. As VP of Worldwide Systems Engineering, he's closer to real-world deployment than the product managers and executives who set roadmap direction. What he sees most often isn't a technology problem. It's an awareness problem.

What the First 90 Days Actually Look Like

When customers first deploy Data Intelligence, the expected conversation is about AI readiness, pipeline automation, and semantic context. What actually happens is simpler and more unsettling: the tool maps the full scope of the customer's data estate, and that scope is almost always larger and messier than anyone anticipated.

Sheth described the sequence clearly. Step one is discovery — understanding what data exists, where it lives, and who can access it. Step two is classification and contextualization — tagging sensitive data, building governance policies, and mapping relationships across systems. Only after those two steps does the conversation about AI pipelines and Data Stream actually make sense, because only then does the customer understand what should and shouldn't be feeding those pipelines.

"Some customers find they actually have to do a lot of data cleanup before they even think about a data pipeline," Sheth said. "You need to make sure the right things are actually going to be going into that pipeline."

The court system demo staged during Wednesday's keynote illustrated the point. A large metropolitan court system — Sheth declined to name it — used Data Intelligence to map its entire data estate across multiple tiers of courts before building any AI models. The reason: too much sensitive data to use any third-party AI service. They needed to understand what they had, wall off what couldn't be touched, and build a secure internal capability — the equivalent of their own Harvey AI — before any pipeline work could begin. Two months after not knowing what data they had, they were extracting value from it.

The Global Manufacturer: What Actually Happened

One of the more detailed customer stories to come out of this week's conference involves a global manufacturer with 80 sites worldwide. Sheth walked through the full arc of that engagement during our conversation, and it's worth being specific about what the case study actually shows — and what it doesn't yet.

The manufacturer was running legacy hyperconverged infrastructure across all 80 sites, with support approaching its end. The security implications were immediate: once the vendor stops issuing updates, every unpatched vulnerability becomes a permanent exposure. That alone was a compelling reason to modernize. But the performance case was equally clear — hyperconverged architecture was struggling to keep pace with production throughput as manufacturing volumes increased.

The engagement ran 60 days, start to finish. The first phase covered the self-assessment, discovery, and transition planning — mapping current-state architecture and modeling what a move to a three-tier architecture based on Everpure FlashArray would look like at scale. The second phase went further: exploring what a video surveillance solution could do for manufacturing quality. The answer, it turned out, was significant. By building AI models around video feeds on the manufacturing line, the company could detect counterfeit parts in real time, reducing defect rates and improving field quality.

One thing worth clarifying: the gross margin expansion Sheth referenced in his keynote presentation is a projection, not a delivered result. The migration plan for 80 global sites is approximately 1 year for phase 1. The business case has been made to the CISO and CIO — and it was strong enough for them to commit — but the results are still being built.

What struck me about the story was Sheth's description of how the engagement was framed. "We anchored on what's most important for them," he said. "Production is money. We really wanted to anchor around what's going to ensure that they have higher production volume, better throughput, and fewer quality issues — versus coming in and saying, hey, we have a really cool technology platform."

That framing — infrastructure partner as business consultant rather than hardware vendor — came up repeatedly throughout our conversation.

The Migration Risk Nobody Talks About

The most honest answer Sheth gave came when I asked about the most common architectural mistake systems engineering teams see when customers try to scale before addressing existing risk.

His answer wasn't what I expected. He didn't describe a technical anti-pattern or a common configuration error. He described organizational inertia. "One of the biggest general challenges is sometimes a customer has been with a certain platform for five years, and they feel like moving to a different provider could end up being a bigger challenge or a bigger risk than staying with the incumbent."

His point is that the decision not to move is itself a risk posture — one that customers often don't examine critically because it feels like the safe default. Everpure's response to that has been to build SLA-backed migration tooling and use Data Intelligence to map what's running on the incumbent environment before any transition begins, making the migration plan more defensible and reducing the perceived risk of change. The goal is to earn trust incrementally, demonstrate value on the customer's existing infrastructure, and let the case for deeper adoption build from there.

"We want to meet them where they're at, we want to earn their trust, and then over time they'll continue to grow with us," Sheth said. The 84 NPS score — independently audited — is the metric he cited as evidence that the approach is working.

The Blueprint as Sequencing Tool

Sheth's framing of the EDC Success Blueprint differs slightly from how it was presented in formal announcements, and the difference is worth surfacing for engineering leaders evaluating whether to engage with it.

Officially, the Blueprint is a structured methodology — self-assessment, maturity guides, dedicated workshop — for moving toward an Enterprise Data Cloud architecture. In practice, Sheth describes it less as a methodology and more as a sequencing tool. Its real value is in helping organizations with 15 competing priorities figure out which ones to tackle first, in an order that's actually achievable given their current state, available resources, and internal political realities.

"If you try to take on too much in any one go, you might not have the level of internal support that you need," he said. "We really want to make sure that we're anchoring on what's most important in terms of our customers' priorities and their business outcomes."

The workshops included with Evergreen subscriptions are on-site, dedicated sessions — not a fee-based add-on. Sheth was clear about that distinction because it affects how customers should think about the commitment being asked of them. Everpure is investing its own systems engineering resources in these engagements, which means the company has a stake in the outcome being real rather than just the contract being signed.

A Different Company Than It Was

Sheth's closing comment was the most direct summary of what this week has actually been about, beyond any individual product announcement: "Everpure is a very different company than where we were a couple of years ago."

He pointed specifically to the company's founder, echoing the same sentiment on stage: the company that built its reputation on FlashArray has expanded into a full-stack data intelligence platform where storage is the foundation, not the ceiling. Data Intelligence, Data Stream, Portworx — these products add value regardless of whether the underlying storage is Everpure's. That's an intentional architectural bet, not a side effect.

"We can now add value no matter where the data resides," Sheth said.

For developers, engineers, and architects deciding whether to pay attention to Everpure's announcements this week, that's the most useful frame. The company is no longer selling storage performance as its primary value proposition. It's selling visibility and control over data that enterprises already have, wherever that data happens to live — with Everpure storage as the preferred home but not the prerequisite for value.

Whether that bet pays off at scale, across 14,500 customers with varying levels of maturity and readiness, is the story that plays out over the next several years.

Everpure (NYSE: P) provided press access to Pure Accelerate 2026.

Why Enterprise AI Keeps Failing — And What Everpure Is Doing About It

Wed, 17 Jun 2026 14:00:00 +0000

For decades, enterprise IT built everything around applications. SAP owned finance. Salesforce owned customer relationships. ServiceNow owned IT operations. Each application came with its own data, its own definitions, and its own version of what words like "customer" or "asset" actually meant.

That model worked well enough — until AI showed up.

AI doesn't work the way applications do. It doesn't care which system owns which data. It needs to reason across everything at once, in real time, with consistent context. And when it tries to do that inside a fragmented, application-centric architecture, it fails. According to a June 2026 IDC survey commissioned by Everpure, 54% of AI proof-of-concept projects never reach production. That's a majority of AI budgets generating zero return on investment.

The root cause, according to IDC's research of more than 1,300 organizations, isn't compute. It isn't talent. It's data. Ninety-four percent of respondents said data quality is important or very important to AI project success. The top contributors to poor data quality are redundant data, multiple storage silos, and obsolete data.

At Pure Accelerate 2026 in Las Vegas, Everpure is making a direct argument that this is an architectural problem — and that fixing it requires inverting 50 years of enterprise IT assumptions.

The Problem with App-Centric Architecture

Charles Giancarlo, Chairman and CEO of Everpure, laid out the historical case in a keynote essay released alongside this week's announcements.

The original vision for large-scale ERP systems was a single semantic framework — one shared definition of "customer," "order," and "asset" across the entire enterprise. That vision never survived contact with real-world IT. A best-of-breed era arrived instead, with specialized applications for every business domain. Each brought its own definitions, its own data copies, and its own ETL pipelines to communicate with everything else.

SaaS accelerated the problem. Every new application brought more fragmentation, more copies, more integration work. The typical enterprise today has hundreds or even thousands of applications, each with its own data set, requiring costly transformation before the data can be used elsewhere.

In the AI era, this isn't just inefficient. As Giancarlo puts it, it's unsustainable.

"Customer" in a CRM means something different than "customer" in a billing system. "Asset" in an ERP is not the same object as "asset" in a supply chain platform. Multiply that across hundreds of applications, and you have a system in which cross-domain meaning must be reconstructed every time it's needed. AI fed on that fragmented, inconsistent data produces inconsistent results. In business, accounting, and R&D, 95% accuracy isn't good enough.

The Data Primacy Model

Everpure's response is what Giancarlo calls data primacy — a fundamental inversion of the hierarchy between applications and data.

In the data primacy model, data becomes the system of record. Applications read from and write to shared data, but they don't own it. Context and semantics travel with the data itself, not embedded inside the application that created it. Governance is enforced at the data layer rather than policed by external software.

This rests on three principles Giancarlo articulated this week:

Semantics must travel with the data. Operational data must carry meaning that can be understood outside the system that created it. This requires building semantic descriptions alongside data relationships that define how a piece of data connects to everything else in the enterprise. Raw data becomes self-describing, contextually rich, and ready for AI to reason over accurately.

Data must be conserved, not copied. Enterprises have been conditioned to duplicate data for every new application, analytics environment, and AI use case. Data primacy inverts that. A single authoritative source of real-time data, governed and semantically described, serves all applications and all agents. One version. One truth.

Governance must be embedded at the data layer. In an app-centric world, access is managed through role-based controls tied to individual applications. In a data primacy world, AI agents operate across multiple systems simultaneously. Access must be contextual and dynamic — driven by the attributes of the data itself, the identity of the requesting agent, and the purpose of the action. Conventional role-based access control simply wasn't designed for that.

What Everpure Announced

To deliver on data primacy, Everpure announced three platform innovations at Pure Accelerate 2026, organized around what the company calls the Enterprise Data Cloud.

Universal Data Intelligence

The centerpiece announcement is Everpure Data Intelligence, formerly known as 1touch.io. It's a data intelligence layer that discovers, classifies, and contextualizes enterprise data at its source — across Everpure storage, public clouds, SaaS applications, third-party storage, and legacy environments, including mainframes.

Data Intelligence delivers three core capabilities. Universal Discovery provides visibility into both structured and unstructured data, regardless of storage format, covering major databases, including SQL Server and Oracle. Automated Governance automatically scans the environment to identify sensitive data — PII, PHI — and tracks lineage to maintain a complete map of the data landscape. AI-Ready Context maps raw data to its real-world business definitions, building a semantics knowledge graph that enables AI agents to understand, query, and act on information across the entire enterprise.

For enterprises deploying AI agents, the knowledge graph matters for a specific reason: accurate, contextually relevant data maximizes response accuracy while dramatically reducing context window sizes and token costs. Small language models fed focused, relevant data are outperforming large language models given raw, fragmented data — a practical reality that flips the assumption that bigger models always win.

Ashish Gupta, GM of Data Management and former CEO of 1touch, shared a concrete customer example during the press conference. A large credit card company was spending 21 person-hours per data subject access request (DSAR), with more than 7,000 requests per day. After implementing Data Intelligence, response time dropped to 30 seconds. Implementation for a mid-sized customer takes roughly a month — setup in week one, ML-based classification in weeks two and three, and knowledge graph development from week four onward.

Data Intelligence is also explicitly designed to work on data you don't store on Everpure. Gupta was direct about the competitive differentiation: most vendors in this space ask you to copy your data into their system. Everpure's approach works with data where it already lives — with more than 175 connectors spanning structured and unstructured data, SaaS, on-premises, cloud, and mainframe environments.

Unified Data Plane

The Unified Data Plane is Everpure's shared storage foundation spanning cloud, core, and edge across every workload type — from data archives and enterprise file to real-time applications, containerized workloads, and large AI factories.

Three new developments expand the Data Plane this week.

FlashArray//XL 190 with Purity Turbo Mode consolidates mission-critical workloads onto a single array with more performance and SLA headroom. Everpure Cloud Azure Native Virtual Machines — generally available in July — decouples storage from compute for storage-heavy workloads in Azure, enabling lift-and-shift migration without refactoring and management through the Azure portal as a first-party service.

Evergreen//One Overdrive, available in Q3 2026, lets organizations burst 25% above their SLA for both performance and capacity, paying only for what they actually use. It addresses a practical operational problem: AI workloads are unpredictable. Permanent capacity upgrades are expensive. Overdrive lets organizations absorb traffic spikes without permanent subscription increases.

Intelligent Control Plane

The Intelligent Control Plane is Everpure's fleet-wide operating model for governing data and automating actions across the entire environment. It turns manual, reactive storage administration into a self-optimizing system.

New and enhanced capabilities announced this week include Open Telemetry support, Fusion Fleet Topology, and two MCP servers — Pure1 MCP Server for intelligent queries and Fusion MCP Server for taking action across the global infrastructure estate.

The most compelling new addition is an agentic performance triage workflow. When high latency is detected, the system automatically creates a ServiceNow ticket, notifies via Slack, assigns a triage agent, validates the root cause against vSphere, and hands off to a human when remediation requires authorization. It's a clean, real-world example of human-in-the-loop agentic operations — automation handles detection and diagnosis, while humans retain control over action.

Also notable: Compliance Monitoring and Remediation, available in preview, automatically detects hardware and software configuration drift and enforces global corporate governance policies. Fusion Compliance and Agentic Triage (available Q4 2026) extends this by using agentic AI to suggest root causes for immediate technical remediation.

EDC Success Blueprint

To help organizations move from vision to practice, Everpure introduced the EDC Success Blueprint — a step-by-step methodology across 10 operational pillars for building and scaling an Enterprise Data Cloud. It starts with a readiness assessment to identify immediate infrastructure and security risks, then maps a clear path toward highly automated, efficient architecture.

Options Technology, a global capital markets infrastructure provider, offers an early proof point. After standardizing on Everpure FlashArray, FlashBlade, Evergreen//One, Pure1, Fusion, and Portworx Enterprise, Options achieved roughly an 80% reduction in storage management workload through Fusion and Pure1 automations, along with up to an 80% performance boost for key workloads.

Where Most Enterprises Actually Are

IDC's research is honest about the gap between where organizations need to be and where they are. Sixty percent of survey respondents said their storage infrastructure requires significant improvements or a total refresh to support AI workloads. Thirty-four percent report that their data scientists spend more than half their time waiting on IT — waiting for datasets to be provisioned, GPU time to be allocated, requests to be fulfilled — rather than building models.

IDC identifies four maturity tiers in its AI Readiness Index: Experimenting, Practitioners, Competent, and Masters. Most enterprises are currently sitting somewhere between Experimenting and Practitioner — AI projects are beginning to reach production, but storage and data access bottlenecks are still hampering GPU utilization and slowing project timelines.

The path to Competent and eventually Master status runs through exactly what Everpure announced this week: structured data governance, purpose-built AI storage infrastructure, and the elimination of data silos through intelligent platforms.

Everpure is making a bet that the storage layer is where this transformation has to happen — not inside any individual application, not inside a data lake that's already out of date, but at the data layer itself, where context and governance can travel with the data wherever it goes.

For enterprise IT teams trying to figure out why their AI pilots keep failing, that argument is worth paying attention to.

Everpure (NYSE: P) provided press access to Pure Accelerate 2026. The IDC research cited in this article was commissioned by Everpure.

Building Research Infrastructure at Scale: Inside Globus's Hybrid SaaS Architecture

Mon, 26 Jan 2026 21:26:10 +0000

Research computing faces a unique architectural challenge: how do you build a platform that works across institutional boundaries, handles petabyte-scale data movement, supports diverse storage systems, and maintains fine-grained security controls, all without requiring users to be infrastructure experts?

Globus, a non-profit service from the University of Chicago, has spent nearly 30 years solving this problem. What started as the Globus Toolkit for grid computing has evolved into a comprehensive platform-as-a-service that now handles 2 petabytes of data transfer daily across 2,600+ institutions in 80+ countries.

The Hybrid Architecture

The key architectural insight behind Globus is its hybrid model. "We can't do that because all the institutions own their own resources," explains Rachana Ananthakrishnan, Executive Director of Globus. "We can't sort of take over those resources. We have to figure out how to add valuable services as a layer to that resource."

This means Globus doesn't centralize data—instead, it orchestrates movement between endpoints while data flows directly over high-speed research networks (often 100-400 Gbps). The architecture consists of three main components:

Local Agents run at institutional sites. The Globus Connect agent handles data operations with plugins for different storage types (POSIX, object stores, tape archives). The Globus Compute agent manages remote function execution on various schedulers (Slurm, PBS, Kubernetes). Action Providers allow integration with custom APIs.

Global Management Services runs as a multi-tenant SaaS. These services handle orchestration, heuristic-based optimization, reliability guarantees, and maintain the security fabric. But crucially, they maintain only control-plane connections—file data moves directly between endpoints.

Security Fabric uses OAuth and OIDC to federate authentication across 2,140+ identity providers. This allows researchers to use institutional credentials without creating new accounts, while maintaining fine-grained authorization across distributed resources.

Platform Capabilities

Globus offers six core capabilities that can be used independently or composed:

Managed Transfer provides fire-and-forget data movement with automatic retry, checksum verification, and progress tracking. "The protocol here continuously sends what are called markers," Ananthakrishnan explains. "So it says I've moved 5 bytes, 10 bytes, 20 bytes, and suddenly there's no marker. Then this keeps retrying with a backoff algorithm."

Data Sharing creates permission overlays on existing storage without staging data. Share with any identity, set time restrictions, and maintain audit logs, all without creating local accounts for collaborators.

Unified Access abstracts heterogeneous storage through an open connector ecosystem. The Community Connector Program allows vendors and users to build plugins for any storage system, while Globus handles authentication, translation, and rate limiting.

Search provides schema-agnostic metadata indexing with dynamic schema detection and fine-grained visibility controls. Think Elasticsearch optimized for research with built-in access control.

Compute brings function-as-a-service to on-prem HPC. Submit Python functions that run on remote systems with federated authentication. One COVID-era drug screening project used this to distribute ML workloads across diverse resources, completing training cycles in 31 seconds, compared with much longer traditional approaches.

Flows offers declarative workflow automation using JSON-based definitions. Event-driven execution with AWS Step Functions under the hood, but extended to securely call outside services and remote systems.

Real-World Performance

The Advanced Photon Source at Argonne National Laboratory demonstrates the platform's capabilities at scale. Multiple beamlines use Globus to automate data movement and processing. "These data services have taken the time to solve a structure from weeks to days and now to hours," notes beamline scientist Darren Sherrell.

For developers, Globus provides comprehensive REST APIs and SDKs. The platform is extensible—you're not locked into the web interface. The Franklin & Marshall College biology department even built custom flows that allow field volunteers to click a button, fill a questionnaire, and trigger automated processing. "It really increased the community participation," Ananthakrishnan notes. "In place of teaching all his volunteers how to do these 10 things, he defined a flow."

Compliance and Security

For protected data, Globus implements NIST 800-53 and 800-171 controls, signs BAAs for HIPAA, and complies with GDPR and data protection agreements. The architecture supports both personal data (visible only to the user) and shared data (visible across institutions) with appropriate access controls.

The security model is shared: Globus controls the platform layer, while institutions control their endpoints. "It's a hybrid SaaS, it's a shared security model," Ananthakrishnan explains. "We do some controls, they have to do some controls."

Getting Started

Globus offers a freemium model, basic features are free for non-profit research. Developers can start with the web interface or dive directly into the APIs. Documentation is available at docs.globus.org, and the platform includes a 90-day trial for qualified projects.

For architects building research platforms, Globus provides a reference architecture for handling distributed data at scale. The hybrid model, federated authentication, and connector ecosystem offer lessons applicable beyond scientific computing to any scenario requiring secure data movement across organizational boundaries.

Australian data firm Arcitecta manages a trillion files in one namespace—adds vector search.

Tue, 07 Oct 2025 11:41:27 +0000

How Arcitecta Built a Data Platform That Scales to Trillions of Files

Most data management platforms break down when you hit the petabyte scale. Arcitecta's Mediaflux doesn't.

The Australian company has been building data infrastructure since 1998. Their current claim sounds impossible: a single customer is managing over one trillion files in a single namespace. Not across multiple systems. One unified view.

I met with the Arcitecta team during the 64th IT Press Tour in New York to understand how they built this.

Built from scratch

Arcitecta writes everything themselves. Their XODB database. Their protocols. Their file systems. All first-principles engineering.

XODB is an XML-encoded object database that sits at the core of Mediaflux. It was created in 2010 and supports objects, geospatial data, time series, and now vector embeddings.

This matters because most file systems separate metadata from the actual file system index. XODB doesn't. Everything lives in the database. This makes finding relationships between data much faster.

The database footprint is minimal. For one deployment managing hundreds of billions of files heading toward trillions, they achieved an average of 75 bytes per node. That's two orders of magnitude more dense than typical file systems.

Protocol agnostic by design

Mediaflux supports NFS, SMB, S3, and SFTP. But here's what makes it different: they wrote all these protocols themselves.

Why does this matter? Because you're not locked into any single vendor's implementation. Princeton University uses Mediaflux to manage 200 petabytes of research data across Dell PowerScale, IBM Spectrum Scale, Dell ECS, IBM Cloud Object Storage, and IBM Diamondback tape libraries. All accessible through the same interface.

The IBM Diamondback integration is particularly interesting. IBM's tape system didn't have native S3 support. Arcitecta built it. Now, researchers at Princeton can access tape-archived data through standard S3 calls.

Real-time collaboration

At NAB 2025, Arcitecta showed something new: Mediaflux Real-Time.

This is a replicated file system that lets someone write a file continuously at location A while someone at location B reads it as it's being written. Latency is 10 to hundreds of milliseconds.

Think live video editing. A camera operator captures footage in one location. An editor on another continent can start working on it immediately. No waiting for uploads to complete.

The system uses Fabric Livewire, Arcitecta's parallel TCP/IP transport layer. It can move data at up to 95% of link speed across continents.

Adding vector search

This year, Arcitecta added vector database support to XODB.

The implementation is straightforward. XODB already stored metadata in what's essentially an XY graph. They added a Z dimension for vectors.

This means you can now search across files, metadata, and vector embeddings in a single query. Dana-Farber Cancer Institute is using this for their AI pipelines. National Film and Sound Archive of Australia uses it with Wasabi AiR for facial recognition and content analysis.

The vector embeddings come from external services. Arcitecta doesn't generate them. They store and index them. This lets you use whatever embedding model makes sense for your data.

Python module on the way

Deployment has been Mediaflux's weak point. The system is powerful but complex to set up.

A Python module is coming in the next few months. This should make integration easier for teams already working in Python.

They're also working on better deployment tools. The goal is to reduce the weeks-to-months timeline for getting a system fully operational.

No capacity-based pricing

Arcitecta charges by concurrent users, not data capacity.

This is unusual. Most storage vendors charge by the terabyte. Arcitecta wants you to store as much data as you need. Their revenue model doesn't penalize you for data growth.

For research institutions generating petabytes of data from new instruments, this pricing model matters.

The technical differentiators

Three things set Mediaflux apart technically:

First, it's in the data path. It's not just a management layer on top of storage. It controls data movement, tiering, and access.

Second, everything runs through XODB. This means you can track every operation—who created what file, when, from where, and who accessed it. Princeton exports 70 million audit events per month.

Third, the system is storage agnostic. You can add new storage at any time without migration. Applications keep working. Users see the same namespace.

Where this matters

Arcitecta's customers are mainly research institutions and archives. Princeton, MIT, Dana-Farber Cancer Institute, Technische Universität Dresden, Imperial War Museum, National Film and Sound Archive of Australia.

These organizations generate massive amounts of unstructured data. Cryo-EM microscopes. Genomics machines. Film archives. War records.

Traditional storage systems can't keep up. And vendor lock-in becomes a real problem when you're managing hundreds of petabytes.

What's next

Beyond the Python module, Arcitecta is expanding vector database capabilities and building more deployment tools.

They're also starting to talk about HPC workloads. Not competing with purpose-built parallel file systems, but handling a broader range of high-performance computing work.

The company has been quietly building this platform for over two decades. Now they're starting to tell people about it.

Komprise transforms unstructured data management for AI, delivering security, governance, and 70% storage cost savings.

Sun, 11 May 2025 21:55:46 +0000

Komprise: Building the Intelligent Data Engine for Enterprise AI

Introduction

In an era where unstructured data growth continues to explode and AI adoption accelerates, organizations face significant challenges managing their data estate. Komprise has evolved from a pure storage optimization company to a comprehensive data management platform that serves as the critical intelligence layer between storage infrastructure and AI initiatives. Growing at 40% year-over-year, this Silicon Valley company is helping enterprises not only reduce storage costs but also prepare their data for AI applications while ensuring proper governance and security.

The Unstructured Data Challenge

Unstructured data now represents 90% of enterprise data and continues to grow exponentially. This creates several critical challenges for organizations:

Storage costs spiraling out of control
Data silos preventing comprehensive visibility
Complexity in moving data across environments
Security and governance concerns, especially with AI

As Kumar Goswami, CEO and co-founder explains: "We're changing the way the world manages data - from storage administration to strategic analytics-driven data management, from data locked away to data securely available when and where it's needed."

Technical Architecture

Komprise's distributed architecture consists of three main components:

Komprise Director
The Director serves as the management console providing unified visibility across storage environments. It offers an intuitive UI for administrators to analyze data, create policies, and initiate workflows.

Komprise Observer
These virtual machines are deployed close to storage resources and handle analysis and data movement. Observers work without agents or stubs and stay out of the hot data path, ensuring no impact on active data access.

Key benefit: Since Observers process data locally, sensitive data never leaves the customer's environment, addressing key security concerns for AI data preparation.

Komprise Cloud
At the heart of the architecture is the cloud-based global file index, which maintains metadata about all managed files across the organization's entire storage footprint. This meta database enables:

Scalable indexing of billions of files
Custom tagging and classification
Policy engine for automation
Data lineage tracking
Granular search capabilities

The AI Data Management Imperative

Recent Gartner research indicates that while only 5% of enterprises will train their own AI models, 95% will use pre-trained models that need enterprise data augmentation. This creates several technical challenges Komprise is addressing:

Data Discovery and Mobilization
Finding relevant data across storage silos and moving it efficiently is critical for AI initiatives. Komprise's platform can search, tag, and move data based on complex criteria without disrupting production workloads.

Sensitive Data Detection
Released in January 2025, this capability enables:
Local scanning for PII and custom patterns using regular expressions

Automated classification and tagging workflows
Policy-based remediation actions
Pre-processing to exclude sensitive data from AI training sets

Tip: Organizations can define custom regex patterns to identify company-specific sensitive data types beyond standard PII elements.

Smart Data Workflow Manager
This tool provides a wizard-driven interface to create end-to-end workflows for AI data preparation:

Search and identify relevant data across silos
Scan for sensitive information
Tag and classify data appropriately
Move data to target locations for AI processing
Maintain audit trails of the entire process

Real-World Impact

The business impact of Komprise's approach is significant across different sectors:

Financial Services
A major financial services firm expanded from an initial 10 petabyte migration project to managing over 100 petabytes with Komprise. Their IT team reported that "Komprise has the most comprehensive migration tool on the market for large-scale global migrations," achieving 27x faster data migrations compared to traditional tools.

Healthcare
A research hospital leveraged Komprise to implement storage-as-a-service with departmental chargeback. This gave research teams direct visibility into their data usage and costs while maintaining centralized governance. When searching for historical medical images, Komprise reduced search time from months to hours, enabling researchers to find 9 specific JFK images among 3 million files in just 1 hour and 53 minutes.

Legal Services
A law firm initially adopted Komprise for Azure migration but expanded to make it a core component of their ransomware protection strategy by reducing their attack surface and securing sensitive client data.

Bridging the Gap Between Storage and AI

What makes Komprise unique is its ability to serve as the data engine for AI initiatives while maintaining its core strength in storage optimization. The platform typically delivers:

70% savings on storage, backup, and cloud costs
27x faster data migrations compared to traditional tools
80% more efficient AI data workflows

Smart Data Ingestion for AI

Komprise recently demonstrated how organizations can safely use corporate data to ground AI models using Retrieval-Augmented Generation (RAG):

Search and identify relevant data across silos
Scan for and exclude sensitive information
Move approved data to cloud containers
Index this data for use with OpenAI and other LLM platforms
Maintain complete audit trails of what data was used

This approach ensures that:

Sensitive data never leaks to public AI models
The organization maintains control over what data is used
The AI benefits from accurate, relevant corporate information
Compliance and governance requirements are met

Conclusion

As organizations continue to grapple with explosive unstructured data growth and the emergence of AI workloads, Komprise's evolution from a data management platform to an AI data engine positions it to address critical enterprise needs. The platform's architecture ensures scalability and performance while maintaining security and governance. For IT and engineering teams, Komprise offers both immediate cost savings through storage optimization and strategic value in preparing data for AI initiatives safely and efficiently. With capabilities like sensitive data detection and smart data workflows, organizations can confidently connect their corporate data to AI systems while maintaining proper governance and security.