Coder Legion - Recent posts tagged hybridcloud

Building Research Infrastructure at Scale: Inside Globus's Hybrid SaaS Architecture

Mon, 26 Jan 2026 21:26:10 +0000

Research computing faces a unique architectural challenge: how do you build a platform that works across institutional boundaries, handles petabyte-scale data movement, supports diverse storage systems, and maintains fine-grained security controls, all without requiring users to be infrastructure experts?

Globus, a non-profit service from the University of Chicago, has spent nearly 30 years solving this problem. What started as the Globus Toolkit for grid computing has evolved into a comprehensive platform-as-a-service that now handles 2 petabytes of data transfer daily across 2,600+ institutions in 80+ countries.

The Hybrid Architecture

The key architectural insight behind Globus is its hybrid model. "We can't do that because all the institutions own their own resources," explains Rachana Ananthakrishnan, Executive Director of Globus. "We can't sort of take over those resources. We have to figure out how to add valuable services as a layer to that resource."

This means Globus doesn't centralize data—instead, it orchestrates movement between endpoints while data flows directly over high-speed research networks (often 100-400 Gbps). The architecture consists of three main components:

Local Agents run at institutional sites. The Globus Connect agent handles data operations with plugins for different storage types (POSIX, object stores, tape archives). The Globus Compute agent manages remote function execution on various schedulers (Slurm, PBS, Kubernetes). Action Providers allow integration with custom APIs.

Global Management Services runs as a multi-tenant SaaS. These services handle orchestration, heuristic-based optimization, reliability guarantees, and maintain the security fabric. But crucially, they maintain only control-plane connections—file data moves directly between endpoints.

Security Fabric uses OAuth and OIDC to federate authentication across 2,140+ identity providers. This allows researchers to use institutional credentials without creating new accounts, while maintaining fine-grained authorization across distributed resources.

Platform Capabilities

Globus offers six core capabilities that can be used independently or composed:

Managed Transfer provides fire-and-forget data movement with automatic retry, checksum verification, and progress tracking. "The protocol here continuously sends what are called markers," Ananthakrishnan explains. "So it says I've moved 5 bytes, 10 bytes, 20 bytes, and suddenly there's no marker. Then this keeps retrying with a backoff algorithm."

Data Sharing creates permission overlays on existing storage without staging data. Share with any identity, set time restrictions, and maintain audit logs, all without creating local accounts for collaborators.

Unified Access abstracts heterogeneous storage through an open connector ecosystem. The Community Connector Program allows vendors and users to build plugins for any storage system, while Globus handles authentication, translation, and rate limiting.

Search provides schema-agnostic metadata indexing with dynamic schema detection and fine-grained visibility controls. Think Elasticsearch optimized for research with built-in access control.

Compute brings function-as-a-service to on-prem HPC. Submit Python functions that run on remote systems with federated authentication. One COVID-era drug screening project used this to distribute ML workloads across diverse resources, completing training cycles in 31 seconds, compared with much longer traditional approaches.

Flows offers declarative workflow automation using JSON-based definitions. Event-driven execution with AWS Step Functions under the hood, but extended to securely call outside services and remote systems.

Real-World Performance

The Advanced Photon Source at Argonne National Laboratory demonstrates the platform's capabilities at scale. Multiple beamlines use Globus to automate data movement and processing. "These data services have taken the time to solve a structure from weeks to days and now to hours," notes beamline scientist Darren Sherrell.

For developers, Globus provides comprehensive REST APIs and SDKs. The platform is extensible—you're not locked into the web interface. The Franklin & Marshall College biology department even built custom flows that allow field volunteers to click a button, fill a questionnaire, and trigger automated processing. "It really increased the community participation," Ananthakrishnan notes. "In place of teaching all his volunteers how to do these 10 things, he defined a flow."

Compliance and Security

For protected data, Globus implements NIST 800-53 and 800-171 controls, signs BAAs for HIPAA, and complies with GDPR and data protection agreements. The architecture supports both personal data (visible only to the user) and shared data (visible across institutions) with appropriate access controls.

The security model is shared: Globus controls the platform layer, while institutions control their endpoints. "It's a hybrid SaaS, it's a shared security model," Ananthakrishnan explains. "We do some controls, they have to do some controls."

Getting Started

Globus offers a freemium model, basic features are free for non-profit research. Developers can start with the web interface or dive directly into the APIs. Documentation is available at docs.globus.org, and the platform includes a 90-day trial for qualified projects.

For architects building research platforms, Globus provides a reference architecture for handling distributed data at scale. The hybrid model, federated authentication, and connector ecosystem offer lessons applicable beyond scientific computing to any scenario requiring secure data movement across organizational boundaries.

HPE unveils agentic AI, smart infrastructure, and developer tools at Discover 2025 conference.

Tue, 24 Jun 2025 18:59:42 +0000

HPE Discover 2025: Agentic AI and Infrastructure Evolution for Developers

As HPE celebrates its 10th anniversary as a standalone company, the tech giant is positioning itself at the forefront of what it calls the "AI-native era." At HPE Discover 2025, the company unveiled a comprehensive suite of innovations that promise to fundamentally change how developers, engineers, and architects approach hybrid cloud infrastructure, AI development, and enterprise operations.

The Dawn of Agentic AI Infrastructure

The headline announcement centers on GreenLake Intelligence, HPE's new agentic AI framework that transforms traditional IT operations from reactive to proactive and autonomous. Unlike conventional AI tools that merely provide recommendations, agentic AI can actually execute actions, analyze results, and iterate on solutions—all while maintaining human oversight.

For developers, this represents a paradigm shift. Instead of manually troubleshooting infrastructure issues or writing complex monitoring scripts, GreenLake Intelligence deploys specialized AI agents that can:

Automatically diagnose and resolve network issues across hybrid environments
Optimize workload performance by intelligently redistributing resources
Predict and prevent failures before they impact applications
Generate contextual insights from the 20 trillion metrics HPE collects globally

Varma Kunaparaju, former CEO of OpsRamp (now part of HPE), emphasized during the briefing: "Our design with GreenLake Intelligence is to help customers fix issues faster, prevent failures earlier, shift focus to innovation and ultimately deliver business outcomes."

Smart Networking That Thinks for Itself

Perhaps nowhere is the power of agentic AI more evident than in HPE Aruba Networking Central's new capabilities. The system demonstrates autonomous problem-solving that would typically require hours of human analysis.

Consider this real-world scenario: When wireless clients experience authentication failures while roaming between access points, the traditional approach requires network administrators to manually investigate alerts, correlate data, and implement fixes. With agentic AI, the system:

Automatically performs blast radius analysis to understand the scope of impact
Identifies root causes by correlating device compatibility with network features
Proposes specific remediation steps with detailed pros and cons analysis
Provides actionable recommendations that can be implemented immediately or scheduled

David Hughes, Chief Technology Officer at HPE Aruba Networking, noted that this technology works "like having a teammate that can work while you're asleep, work on problems, and when you arrive in the morning, have those proposed answers there."

Unified Cloud Operations with CloudOps Software Suite

For developers managing complex hybrid environments, HPE introduced the CloudOps Software suite, which unifies three critical capabilities:

HPE Morpheus Enterprise Software for workload orchestration and management
HPE OpsRamp Software for comprehensive observability and monitoring
HPE Zerto Software for data protection and disaster recovery

This unified approach addresses a common pain point for development teams: the complexity of managing disparate tools across hybrid cloud environments. The suite provides a single control plane for orchestrating applications, monitoring performance, and ensuring data resilience across on-premises, edge, and public cloud infrastructure.

AI Factories: From Concept to Production

HPE's expanded AI factory portfolio addresses the full spectrum of AI development needs. For developers and engineering teams, several offerings stand out:

Enterprise-Ready AI with Private Cloud AI

The next-generation Private Cloud AI introduces several developer-friendly features:

Federated architecture that allows GPU resource pooling across multiple generations of hardware
Multi-tenancy capabilities with GPU quotas and role-based access control
Air-gapped deployment options for sensitive development environments
Integration with NVIDIA AI Blueprints for rapid AI application development

Storage That Understands AI Workloads

The HPE Alletra Storage MP X10000 represents a significant advancement for AI-intensive applications. By integrating Model Context Protocol (MCP) servers, the storage system can:

Communicate directly with AI agents to optimize data pipelines
Provide intelligent metadata enrichment for unstructured data
Enable conversational data access through natural language interfaces
Accelerate AI training and inference workflows

As Sherry Williams, SVP and GM of Private Cloud & Flex Solutions, explained: "This is how you can now talk to your data with HPE Alletra using an agentic co-pilot in real time."

Developer Ecosystem and Partnership Expansions

HPE significantly expanded its developer ecosystem through several key partnerships:

Enhanced Data Protection Integration

The strengthened partnerships with Commvault and Veeam provide developers with robust data protection options:

Veeam integration with HPE Morpheus VM Essentials for comprehensive backup and recovery
Commvault offering HPE Zerto Software to extend disaster recovery capabilities
Native backup support for virtualized development environments

Expanded AI Use Case Library

Through the Unleash AI program, HPE now supports over 75 AI use cases with 26 new partners, including:

Rapid application development tools for faster prototyping
Scalable data science workflows for ML/AI projects
Advanced model development platforms
Enterprise-grade AI frameworks for production deployments

Real-World Developer Impact

The practical implications for development teams are substantial. Traditional infrastructure management often consumes 30-40% of IT resources on routine maintenance and troubleshooting. With agentic AI handling these tasks autonomously, development teams can redirect focus toward innovation and feature development.

The zero-percent financing program for CloudOps software and new assessment tools like CloudPhysics Plus lower barriers to adoption, making it easier for organizations to modernize their development infrastructure without significant upfront investment.

Looking Ahead: The Infrastructure Evolution

HPE's vision extends beyond current capabilities. The integration of agentic AI across compute, storage, and networking creates an intelligent infrastructure layer that adapts to application needs in real-time. For developers, this means:

Infrastructure that self-optimizes based on application behavior
Predictive scaling that anticipates demand patterns
Automated security posture management across hybrid environments
Intelligent resource allocation that maximizes performance while minimizing costs

The Bottom Line for Developers

HPE Discover 2025 signals a fundamental shift toward infrastructure that actively participates in the development lifecycle rather than simply hosting applications. The combination of agentic AI, unified management platforms, and AI-ready storage creates an environment where developers can focus on building innovative solutions rather than managing underlying infrastructure complexity.

As Antonio Neri, HPE's President and CEO, noted during the briefing: "HPE is reimagining hybrid IT as only we can do, catapulting organizations from the era of hybrid complexity to the era of agentic-AI-powered cloud operations."

For development teams, this represents an opportunity to fundamentally rethink how they approach infrastructure, moving from reactive management to proactive optimization powered by intelligent automation.