Coder Legion - Recent posts tagged filestorage

New York startup replaces three media tools with one platform using FUSE, S3, and custom AI models.

Mon, 13 Oct 2025 13:20:57 +0000

How Shade Built a File System That Actually Works for Remote Creative Teams

Media production teams have a storage problem. They use Lucid Link for file streaming, Frame.io for review and approval, and Iconik for asset management. That's three subscriptions, three login screens, and constant file transfers between systems.

Shade thinks this is ridiculous. The New York-based startup built a single platform that handles ingestion, storage, review, and delivery. After raising $5 million from General Catalyst and Signal Fire, they now serve 94 customers and project $10 million in revenue next year.

I recently met with Shade during the 64th IT Press Tour.

The Technical Stack

Shade's architecture sits on top of S3-compatible storage. Customers can bring their own Wasabi, AWS, or Google Cloud Storage buckets, or use Shade's managed storage. Either way, the raw files stay in object storage while Shade's metadata service runs separately on AWS.

The desktop application uses FUSE (Filesystem in Userspace) on Mac and Linux, with similar technology on Windows. When you mount a Shade workspace, it appears as a local drive. Behind the scenes, the FUSE layer intercepts file system calls and translates them into S3 requests.

Brandon Fan, Shade's co-founder and CEO, explained the approach: "We have a CDN infrastructure in front of S3 that speeds up access to various locations. We also have local upload capability—it uploads to the nearest server, then syncs to the origin server."

This architecture supports parallel chunk-based uploads and downloads. The result is fast enough that teams can stream 4K video without downloading entire files first.

Custom AI for Massive Scale

The AI components required more custom work than expected. Shade uses OpenAI's vision models for metadata generation—things like identifying shot types, lighting conditions, and visible logos. But the search engine is completely custom.

"We had to build our own semantic search," Fan said. "Most traditional models are too slow for millions of assets. We took open-source vision models and distilled them to run on CPU infrastructure instead of GPUs. This keeps costs down while letting users search through hundreds of thousands of assets."

The search works across images, videos, and documents. Type "people playing cards in a park" and it finds that exact scene. The system also handles transcription (via Assembly AI) and facial recognition.

For clustering and organizing large asset libraries, Shade built another custom algorithm. Standard clustering falls apart at scale, so they engineered a solution that handles the volume their customers generate.

File Locking and Conflicts

Creative applications like Premiere Pro have built-in project locking. For everything else, Shade creates conflict files when two users modify the same asset. If you rename a folder while someone saves a file into it, Shade preserves both actions and flags the conflict.

The system maintains a multi-tenant database that stores all metadata, permissions, and file relationships. When a client requests a file, it queries this service to get the S3 URL, then streams the data from object storage.

Integration Points

Shade offers webhook-based automation through a Zapier-style interface. Teams connect Shade to their project management tools, CRMs, or custom applications. The API exposes all metadata and file operations.

The platform also supports SSO via SAML, SCIM provisioning, and audit logging. Permissions work at three levels: workspace, individual users, and link-based sharing. You can share files with external collaborators through password-protected links that expire on a schedule.

Developer Background

Fan comes from an information retrieval research background, publishing papers on search technology before the transformer era. His co-founder Emerson, the CTO, is a roboticist who built the distributed file system infrastructure.

The team includes engineers from Lucid Link and Frame.io—people who know the competitive landscape and customer pain points firsthand. Most of the 15-person team works from the New York office to maintain tight collaboration between engineering and product.

Market Position

Frame.io sold to Adobe for $1.2 billion in 2021. Iconik raised $80 million. Shade raised $5 million and competes directly with both.

The cost advantage matters. A team using 10 seats and 10TB on Frame.io plus another 10TB on Lucid Link pays significantly more than the equivalent Shade setup. Fan claims Shade typically runs 30% cheaper than the combined stack it replaces.

Sales cycles run fast—usually a seven-day trial followed by contract signing. Procurement and legal reviews take longer, but the technical evaluation wraps quickly. Most deals close within a month of first contact.

The current average contract value sits around $10,000 to $15,000 annually for a 10-user, 25TB setup. Growth comes primarily through inbound traffic from Google searches and customer referrals.

What's Next

Shade plans to double the team this year, with most new hires in engineering. The roadmap includes sub-clipping (searching within video clips), expanded document workflows, and an app marketplace where third parties can build on the platform.

The vision echoes what Salesforce did for CRM and Notion did for documents—become the default system of record, but for files. Whether that happens depends on execution. But the technical foundation is solid, and the team has proven they can ship fast.

For engineering teams tired of duct-taping together file storage solutions, Shade offers a cleaner option. One platform, one API, one place where files live and teams collaborate.

Australian data firm Arcitecta manages a trillion files in one namespace—adds vector search.

Tue, 07 Oct 2025 11:41:27 +0000

How Arcitecta Built a Data Platform That Scales to Trillions of Files

Most data management platforms break down when you hit the petabyte scale. Arcitecta's Mediaflux doesn't.

The Australian company has been building data infrastructure since 1998. Their current claim sounds impossible: a single customer is managing over one trillion files in a single namespace. Not across multiple systems. One unified view.

I met with the Arcitecta team during the 64th IT Press Tour in New York to understand how they built this.

Built from scratch

Arcitecta writes everything themselves. Their XODB database. Their protocols. Their file systems. All first-principles engineering.

XODB is an XML-encoded object database that sits at the core of Mediaflux. It was created in 2010 and supports objects, geospatial data, time series, and now vector embeddings.

This matters because most file systems separate metadata from the actual file system index. XODB doesn't. Everything lives in the database. This makes finding relationships between data much faster.

The database footprint is minimal. For one deployment managing hundreds of billions of files heading toward trillions, they achieved an average of 75 bytes per node. That's two orders of magnitude more dense than typical file systems.

Protocol agnostic by design

Mediaflux supports NFS, SMB, S3, and SFTP. But here's what makes it different: they wrote all these protocols themselves.

Why does this matter? Because you're not locked into any single vendor's implementation. Princeton University uses Mediaflux to manage 200 petabytes of research data across Dell PowerScale, IBM Spectrum Scale, Dell ECS, IBM Cloud Object Storage, and IBM Diamondback tape libraries. All accessible through the same interface.

The IBM Diamondback integration is particularly interesting. IBM's tape system didn't have native S3 support. Arcitecta built it. Now, researchers at Princeton can access tape-archived data through standard S3 calls.

Real-time collaboration

At NAB 2025, Arcitecta showed something new: Mediaflux Real-Time.

This is a replicated file system that lets someone write a file continuously at location A while someone at location B reads it as it's being written. Latency is 10 to hundreds of milliseconds.

Think live video editing. A camera operator captures footage in one location. An editor on another continent can start working on it immediately. No waiting for uploads to complete.

The system uses Fabric Livewire, Arcitecta's parallel TCP/IP transport layer. It can move data at up to 95% of link speed across continents.

Adding vector search

This year, Arcitecta added vector database support to XODB.

The implementation is straightforward. XODB already stored metadata in what's essentially an XY graph. They added a Z dimension for vectors.

This means you can now search across files, metadata, and vector embeddings in a single query. Dana-Farber Cancer Institute is using this for their AI pipelines. National Film and Sound Archive of Australia uses it with Wasabi AiR for facial recognition and content analysis.

The vector embeddings come from external services. Arcitecta doesn't generate them. They store and index them. This lets you use whatever embedding model makes sense for your data.

Python module on the way

Deployment has been Mediaflux's weak point. The system is powerful but complex to set up.

A Python module is coming in the next few months. This should make integration easier for teams already working in Python.

They're also working on better deployment tools. The goal is to reduce the weeks-to-months timeline for getting a system fully operational.

No capacity-based pricing

Arcitecta charges by concurrent users, not data capacity.

This is unusual. Most storage vendors charge by the terabyte. Arcitecta wants you to store as much data as you need. Their revenue model doesn't penalize you for data growth.

For research institutions generating petabytes of data from new instruments, this pricing model matters.

The technical differentiators

Three things set Mediaflux apart technically:

First, it's in the data path. It's not just a management layer on top of storage. It controls data movement, tiering, and access.

Second, everything runs through XODB. This means you can track every operation—who created what file, when, from where, and who accessed it. Princeton exports 70 million audit events per month.

Third, the system is storage agnostic. You can add new storage at any time without migration. Applications keep working. Users see the same namespace.

Where this matters

Arcitecta's customers are mainly research institutions and archives. Princeton, MIT, Dana-Farber Cancer Institute, Technische Universität Dresden, Imperial War Museum, National Film and Sound Archive of Australia.

These organizations generate massive amounts of unstructured data. Cryo-EM microscopes. Genomics machines. Film archives. War records.

Traditional storage systems can't keep up. And vendor lock-in becomes a real problem when you're managing hundreds of petabytes.

What's next

Beyond the Python module, Arcitecta is expanding vector database capabilities and building more deployment tools.

They're also starting to talk about HPC workloads. Not competing with purpose-built parallel file systems, but handling a broader range of high-performance computing work.

The company has been quietly building this platform for over two decades. Now they're starting to tell people about it.