Coder Legion - Recent posts tagged cloudstorage

Zettalane's Cloud Storage Architecture: Object Storage Performance Without the Price Tag

Sun, 01 Feb 2026 01:52:24 +0000

Zettalane combines ZFS with object storage to deliver 8.14 GB/s throughput at 70% lower cost than AWS EFS. Terraform deployment included.

The Problem with Cloud File Storage

Cloud file storage costs don't make sense. AWS EFS costs $0.30 per GB per month. That's $360,000 annually for 100 TB. Block storage (EBS gp3) costs $0.08 per GB per month but lacks the shared file access that applications require.

The performance issues run deeper than pricing. Most managed file services throttle per-client throughput even when aggregate bandwidth looks good. You also can't access advertised performance until you reach 100 TB of capacity.

What Zettalane Built

Supramani (Sam) Sammandam presented Zettalane's approach at the 66th IT Press Tour in January 2026. The company offers two products:

MayaNAS handles high-throughput file storage using a hybrid ZFS architecture. Metadata resides on NVMe SSDs, while bulk data is written directly to object storage (S3, GCS, Azure Blob). The system delivers 4 GB/s throughput per node with active-active HA.

MayaScale provides ultra-low-latency block storage using local NVMe SSDs via NVMe-over-Fabrics. It delivers 2.3 million IOPS with 129 microsecond latency on GCP.

The Technical Architecture

MayaNAS uses ZFS special VDEVs to separate metadata from data blocks. Files smaller than 128KB (config files, job parameters) stay on NVMe for low-latency access. Large sequential files go straight to object storage.

This is significant for workloads like AI training jobs. The job scheduler reads small config files from NVMe storage. When the job runs, it streams large training datasets directly from object storage at full bandwidth.

Traditional cloud file systems typically cache everything via NVMe or stage data before moving it to object storage. That approach wastes the object storage backend's 200 Gbps bandwidth capability.

Zettalane wrote a Go program called objbacker.io that acts as a native ZFS VDEV for object storage. It uses vendor SDKs (AWS, GCP, Azure) to handle the object storage API calls. When ZFS gets a 1MB write from an NFS client, objbacker.io sends that full 1MB block directly to object storage. Smaller blocks go to the local NVMe VDEV.

Performance Numbers

In testing on GCP with two n2-standard-48 instances (48 vCPU, 192GB RAM each), MayaNAS delivered:

8.14 GB/s concurrent read throughput (active-active HA, both nodes)
6.2 GB/s concurrent write throughput (direct I/O, 1MB blocks)

The test used 10 concurrent FIO jobs, 23GB files per job (230GB total to defeat cache), 300-second sustained runs, and NFS v4 with nconnect=16.

MayaScale performance on GCP using older Y2 family instances:

2.3 million IOPS (peak at QD64)
129 microseconds latency (at QD1)
19x faster than GCP PD Extreme

The Y2 instances cost less than newer Titanium SSD instances while delivering similar performance. Zettalane's Terraform modules automatically select cost-effective instance types.

Deployment

Both products can be deployed via Terraform in about two minutes. The Terraform module handles:

Instance provisioning with optimal network settings
VIP configuration (ENI on AWS, IP alias on GCP, custom routes on Azure)
Object storage bucket creation and authentication
ZFS pool creation and NFS export configuration
Active-active HA setup with automatic failover

You run terraform apply and get a working NFS share with the mount command in the output.

Multi-Cloud Support

The same architecture runs on AWS, GCP, and Azure. Zettalane abstracts the networking differences:

Component	AWS	Azure	Google Cloud
Instance	c5.xlarge	D4s_v4	n2-standard-4
Block Storage	EBS gp3	Premium SSD	pd-ssd
Object Store	S3	Blob Storage	GCS
VIP Migration	ENI attach	LB health probe + Custom route	IP alias
Deployment	CloudFormation	ARM Template	Terraform

The Terraform modules use the same ZFS configuration and cluster setup across all three clouds.

MayaScale Architecture

MayaScale pools local NVMe SSDs across instances using either Linux MD RAID-1 or ZFS zpool mirroring. Both modes provide active-active HA with server-side synchronous replication.

The server-side approach matters. Competing solutions use client-side replication, where the client writes to both storage nodes and waits for ACKs from both. That doubles network traffic and puts mirror management logic on the client.

MayaScale clients write to the storage node once. The storage node handles mirroring internally. The client sees a single write and a single ACK. This cuts network traffic in half and eliminates client overhead.

The system supports three protocols:

NVMe-over-TCP (default)
NVMe-over-RDMA (validated on cloud)
iSCSI

Zettalane validated RDMA on public cloud infrastructure with 30-40 microsecond latency using InfiniBand. They expect to announce public RDMA support soon.

FSX Mode

MayaScale includes an FSX mode that uses ZFS zpool mirroring instead of MD RAID-1. This provides OpenZFS-based NFS file storage for clouds that don't offer AWS FSx equivalents (such as GCP and Azure).

ZFS mirroring copies only written data blocks, not entire disks. This makes it more efficient than MD RAID-1 for file workloads.

PostgreSQL Performance

MayaScale delivered 96,000 TPS on PostgreSQL pgbench using server-side RAID-1 HA. That compares to 60,000-70,000 TPS for AWS Aurora.

Pricing Model

Zettalane charges per vCPU-hour with transparent pricing. No per-GB charges, no per-IOPS charges. Cloud infrastructure costs are separate and billed by the cloud provider.

The software runs in your private VPC with no call-home requirement. Zettalane doesn't monitor your deployment or collect telemetry.

What's Next

The roadmap includes:

Kubernetes CSI driver for container persistent volumes
Cloud-native Lustre with MayaNAS as MDS/OSS backend
pNFS FlexFiles support for scale-out NFS

Available Now

Both products are live on AWS, GCP, and Azure marketplaces. You can deploy via the marketplace (CloudFormation or ARM templates) or directly with Terraform modules.

Zettalane targets developers and small-to-medium businesses that need cloud storage but can't justify the minimum 100TB commitments required by most enterprise solutions. You can start with terabyte-scale deployments and scale up as needed.

The company was founded in 2018 and has a team of about 10 people. Sam previously created MAYASTOR, an early software-defined storage platform for iSCSI/FC SAN in 2007.

The Storage Paradox: Why Your 1TB Smartphone Still Feels Ful

Thu, 02 Oct 2025 15:50:53 +0000

Remember the dreaded “Storage Almost Full” notification? It’s a message that induces a unique kind of modern anxiety. A decade ago, with a modest 16GB phone, this was an expected inconvenience. But today, even with devices boasting storage capacities that rival high-end laptops, that same notification continues to plague users.

We are living in the golden age of mobile storage, yet we constantly feel on the brink of digital destitution. This is the great paradox of mobile storage: as capacity expands exponentially, our perception of sufficiency remains stubbornly unchanged. Despite manufacturers arming us with terabytes, why do we still feel like we're running out of space?

A Terabyte in Your Pocket: The Unprecedented Growth of Mobile Storage

To appreciate the scale of this paradox, consider the rapid evolution of our devices. What was once a luxury feature has become a specification battleground, with storage capacities skyrocketing in just over a decade.

2014: The standard-bearer smartphone offered just 16GB of internal, often non-expandable, memory.
2020: The market diverged, with the average Android device featuring 95.7GB and iOS devices averaging 140.9GB.
2024: The industry average exploded to an impressive 384GB, marking a staggering 2300% increase from 2014.
2025: In a clear signal of market trends, Apple's iPhone 17 is projected to eliminate the 128GB tier, establishing 256GB as the new baseline.

This data paints a clear picture of supply. But to understand the paradox, we must look at the ever-increasing demand.

The Digital Deluge: What's Consuming Our Device Storage?

The feeling of insufficient space isn't an illusion. It's the result of a perfect storm of technological advancement, software evolution, and human behavior.

Our Insatiable Appetite for High-Resolution Media
Modern smartphones are not just communication tools; they are professional-grade creative studios. This power comes at a cost measured in gigabytes.
- Cinematic Video: Capturing video in 4K or 8K—now a standard flagship feature—consumes anywhere from 200MB to several gigabytes per minute.
- Professional Photography: Pro-level photography, particularly in uncompressed RAW formats, produces images that can easily exceed 100MB each.
- Offline Entertainment: Streaming services like Netflix and Spotify encourage downloading vast libraries of content for offline access, quickly turning entertainment into a significant storage liability.
The Unseen Weight of Modern Applications
The apps we use daily have undergone a significant transformation. Simple utilities have evolved into feature-rich ecosystems, and this complexity translates directly into storage consumption. Known as "app bloat," this phenomenon includes not only the initial download size but also the silent accumulation of cache files, user data, and temporary assets that can dwarf the app's original footprint.
The 'Storage Tax': Operating Systems and Bloatware
A significant portion of your phone's advertised storage is unavailable from the moment you unbox it. Modern operating systems are incredibly complex, requiring tens of gigabytes for core functions, security features, and system services. Compounding this is the issue of pre-installed applications (bloatware) that cannot be removed, permanently occupying valuable digital real estate.
The Hoarder in All of Us: Digital Clutter and Redundancy
Our digital habits play a crucial role. We save multiple copies of the same photo across different apps, download files we never open, and hold onto years of message history. This reluctance to delete is deeply psychological, driven by an emotional attachment to our digital memories and a "just in case" mentality that leads to massive data accumulation.

Beyond the Gigabytes: The Psychology of 'Storage Anxiety'

This issue transcends mere technical limitations; it has a tangible psychological impact. "Storage anxiety" has become a prevalent condition among users who, regardless of their available space, compulsively monitor their usage. This behavior is driven by two key fears:

The Fear of Running Out: The anxiety that your device will fail you at a critical moment—unable to capture a priceless photo or download an important file—prompts obsessive management.
The Perceived Value of Space: In the consumer mindset, more storage is equated with better performance, a longer device lifespan, and superior value, even if one's actual usage patterns don't justify the need for it.

Escaping the Paradox: The Future of Data Management

While manufacturers will undoubtedly continue to push the boundaries of physical storage, the true solution lies in a smarter, more holistic approach to data management.

Seamless Cloud Integration: The future is not just about local storage, but about intelligent, on-demand synchronization with cloud services. As this integration becomes more seamless, the line between local and remote data will blur, significantly reducing the burden on the device itself.
Efficient and Lean App Design: The onus is on developers to prioritize optimization. By building leaner applications, managing cache more effectively, and minimizing resource consumption, they can deliver powerful experiences without commandeering a user's entire storage capacity.
Empowering Users with Education: Ultimately, fostering better "digital hygiene" is key. Educating users on effective data management—from clearing caches and deleting duplicates to utilizing cloud offloading—can transform their relationship with their device's storage.

Conclusion: A New Paradigm for Digital Space

The endless cycle of increasing storage capacity is a direct response to the escalating demands of our digital lives. However, simply adding more gigabytes is a temporary fix, not a permanent solution. The paradox of mobile storage reveals that technology alone cannot solve a problem so deeply intertwined with software design and human behavior.

To truly feel like we have "enough" space, we need a paradigm shift. This requires a concerted effort from hardware manufacturers, software developers, and users alike—a move towards an ecosystem where storage is not just vast, but intelligent, efficient, and effortlessly managed.

Graid's SupremeRAID uses GPU acceleration to eliminate storage bottlenecks, delivering 28M IOPS for developers.

Mon, 07 Jul 2025 10:33:49 +0000

Graid Technology Revolutionizes Storage Performance with GPU-Accelerated RAID for Developers

The storage landscape for high-performance computing and AI development is undergoing a fundamental transformation. Traditional CPU-based RAID solutions have long created bottlenecks that limit application performance, forcing developers to architect around storage constraints rather than focusing on optimal code design. Graid Technology's SupremeRAID™ represents a paradigm shift that developers have been waiting for: a GPU-accelerated RAID that eliminates these performance limitations.

Breaking Free from CPU-Based Storage Constraints

For decades, developers working on data-intensive applications have faced a fundamental challenge: storage performance that couldn't keep pace with computational demands. Traditional RAID controllers consume valuable CPU cycles and create throughput bottlenecks that limit the effectiveness of multi-threaded applications, real-time analytics systems, and AI training workloads.

SupremeRAID™ addresses this limitation by offloading RAID operations entirely to dedicated GPU hardware. This "out-of-path" architecture means storage operations no longer compete with application logic for CPU resources, fundamentally changing how developers can approach performance-critical applications.

The results speak for themselves. Graid's flagship solution achieves up to 28 million IOPS and 260GB/s throughput with a single card supporting up to 32 native NVMe drives. Perhaps more importantly for developers, the company reports achieving over 95% of raw NVMe performance in production environments—performance that was previously impossible with traditional RAID solutions.

Game-Changing Performance for AI Development

AI and machine learning developers face unique storage challenges. Training large models requires massive datasets that must be continuously fed to GPUs at tremendous speeds. Any storage bottleneck directly translates to expensive GPU idle time and extended training cycles.

SupremeRAID AE (AI Edition) specifically addresses these challenges with features such as GPUDirect Storage (GDS) support and an Intelligent Data Offload Engine. GDS enables direct NVMe-to-GPU memory transfers, eliminating CPU bottlenecks. The Intelligent Data Offload Engine can put the RAID kernel to sleep during intensive GPU computation phases, freeing up additional GPU resources for AI workloads.

For developers building AI training pipelines, this means they can finally design systems where storage performance matches computational capability. No more architectural compromises to work around storage limitations; just pure performance that scales with your data requirements.

Simplifying Development with Software-Defined Architecture

Traditional RAID controllers require developers to understand complex hardware configurations and work within rigid performance constraints. SupremeRAID™'s software-defined approach provides flexibility that hardware solutions cannot match.

The platform supports seamless scaling across multiple GPU servers through features like NVMe-over-Fabrics (NVMe-oF). Developers can design distributed applications that treat storage as a unified resource pool rather than managing individual server limitations. A single SupremeRAID™ installation can provide RAID protection across multiple servers, dramatically simplifying application architecture.

Integration with existing developer workflows is straightforward. The solution includes RESTful APIs for modern IT automation and supports popular clustering storage systems, such as BeeGFS, Lustre, and Ceph, without requiring data migration.

Real-World Developer Impact

The practical implications for developers are substantial. Consider real-time analytics applications that previously required careful buffering strategies to accommodate storage latency. With SupremeRAID™, developers can design more responsive systems that process data streams without artificial delays.

Game development studios leveraging real-time asset streaming can eliminate loading bottlenecks that previously required complex content management strategies. Scientific computing applications can process larger datasets in memory without the traditional tradeoffs between storage speed and data protection.

Financial trading systems benefit enormously from the ultra-low latency capabilities. TelSwitch, a legal data analytics company, has deployed SupremeRAID™ across multiple systems specifically for processing massive datasets with small block sizes, precisely the kind of workload that traditional RAID struggles with.

Enterprise-Grade Reliability for Mission-Critical Code

Despite its performance focus, SupremeRAID™ maintains enterprise-grade data protection features that developers can rely on. The solution supports RAID 6 configurations, protecting against multiple drive failures while maintaining RAID 10-level performance characteristics.

Advanced features, such as journaling, bad block detection, and extended retry mechanisms, enable developers to architect systems with confidence in data integrity. The US Department of Defense has deployed SupremeRAID for edge computing applications that require "military-grade" reliability standards.

The Future of High-Performance Development

NVIDIA CEO Jensen Huang's recent statement, "for the very first time, your storage system will be GPU-accelerated," validates what Graid Technology has been building. As AI workloads become increasingly central to application development, storage architectures that match GPU performance capabilities become essential.

For developers building the next generation of data-intensive applications—whether AI training systems, real-time analytics platforms, or high-performance scientific computing solutions—SupremeRAID™ represents more than just faster storage. It's the foundation for applications that were previously impossible to build efficiently.

The era of designing around storage limitations is coming to an end. With GPU-accelerated RAID, developers can finally focus on building better software rather than working around infrastructure constraints.

Lucidity AutoScaler automates cloud storage management, cutting costs 70% and eliminating manual provisioning tasks.

Tue, 03 Jun 2025 22:24:00 +0000

Lucidity's AutoScaler Eliminates Storage Headaches for Developers With Zero-Downtime Automation

For developers working in cloud environments, few things are more frustrating than storage management bottlenecks that interrupt development workflows. Manual disk provisioning, capacity planning guesswork, and downtime-prone scaling operations have long plagued engineering teams trying to focus on building applications rather than managing infrastructure. Lucidity's breakthrough AutoScaler technology promises to eliminate these pain points entirely.

The Developer Pain Points Lucidity Solves

According to Vatsal Rastogi, Lucidity's Co-Founder, organizations typically waste 50-70% of their provisioned disk space due to over-provisioning—essentially paying three times what they actually need. For developers, this translates to several critical problems that impact daily productivity:

Manual Provisioning Hell: Traditional storage management requires developers to estimate future capacity needs, often leading to either insufficient storage that causes application failures or massive over-provisioning that wastes budget. As Rastogi demonstrated, "Capacity provisioning is a guesstimate" that forces teams into reactive firefighting mode.

Downtime Dependencies: Perhaps most frustrating for developers is that shrinking storage volumes requires downtime—something that's simply unacceptable in modern development workflows. This limitation forces teams to live with over-provisioned storage indefinitely, as the operational cost of downsizing isn't worth the disruption.

Performance Impacts: Manual expansion processes don't just require downtime—they often impact application performance during scaling operations, creating unpredictable slowdowns that can affect user experience and development testing.

The AutoScaler Developer Experience

Lucidity's solution fundamentally changes how developers interact with storage infrastructure. The AutoScaler operates as a lightweight agent that runs on virtual machines, continuously monitoring storage metrics and making real-time adjustments without any developer intervention.

True Zero-Ops Experience: Once deployed, developers literally forget about storage management. The system maintains 75-80% disk utilization automatically, compared to the industry average of 30%. This optimization happens transparently, allowing developers to focus entirely on application logic rather than infrastructure concerns.

Sub-Minute Response Times: When applications need more storage, AutoScaler expands capacity within one minute with zero downtime. This responsiveness means developers never hit storage walls that interrupt development or deployment workflows.

Revolutionary Shrinking Capability: Most impressively, Lucidity has solved the industry's biggest technical challenge, live shrinking of storage volumes without downtime. This breakthrough means storage automatically rightsizes as application needs change, preventing the cost accumulation that traditionally plagues cloud projects.

Real-World Developer Impact

The numbers from Lucidity's customer case studies paint a compelling picture for development teams:

Major US Airline: Automated 7,500 provisioning tasks that previously required manual intervention, reclaiming hundreds of developer hours. The 77% cost reduction freed up budget for actual feature development rather than infrastructure overhead.

Big 4 Accounting Firm: Eliminated 51,000 manual provisioning tasks during peak tax season, allowing their development team to focus on application optimization rather than storage firefighting.

Leading IT Management Company: Achieved $1.5M in savings while managing over 1 petabyte of data, with 435,000 automated operations that would have otherwise consumed significant developer time.

Technical Integration for Development Teams

For developers concerned about integration complexity, Lucidity's approach is refreshingly simple. The system works by:

Agent-Based Architecture: A lightweight agent deploys on each VM, communicating storage metrics to Lucidity's AutoScaler service. This architecture requires no changes to application code or deployment pipelines.

Multi-Cloud Compatibility: Whether your applications run on AWS EBS volumes, Azure Managed Disks, or Google Cloud Persistent Disks, AutoScaler works transparently across all major cloud providers.

Zero Application Changes: Unlike other storage optimization solutions that require application modifications or specific file system configurations, Lucidity operates at the infrastructure layer without touching application logic.

The Business Impact Developers Care About

Beyond the technical benefits, AutoScaler delivers outcomes that matter to development organizations:

Budget Predictability: With automatic optimization reducing storage costs by 50% on average, development teams can better predict infrastructure spending and allocate more budget to innovation rather than waste.

Reduced Operational Overhead: Teams report reclaiming hundreds of hours previously spent on manual storage management, time that can be redirected to feature development and technical debt reduction.

Improved Application Performance: The 2x IOPS performance gains that result from optimized storage allocation directly benefit application responsiveness and user experience.

Developer-Friendly Implementation

What sets Lucidity apart for development teams is the implementation simplicity. As Raj Dutt, VP of Marketing, emphasized during the presentation, "Once you do a one-time onboarding activity, you're through. All you care about is the efficiency number."

The onboarding process involves:

Simple account integration through cloud provider APIs
One-click agent installation across VM instances
Automatic disk discovery and optimization recommendations
Transparent ongoing management with detailed monitoring dashboards

For developers building modern applications that demand elastic storage capabilities, Lucidity's AutoScaler represents a fundamental shift from reactive storage management to proactive automation. By eliminating the traditional tradeoffs between cost, performance, and operational overhead, it allows development teams to focus on what they do best—building exceptional applications.

The technology's ability to deliver both immediate cost savings and long-term operational efficiency makes it particularly compelling for development organizations looking to modernize their cloud infrastructure without adding complexity to their development workflows.