AWS AgentCore's Multi-Account Architecture: How Separate Inference and Tool Gateways Enforce Agent Governance at Scale

AWS just published a reference blueprint for AgentCore, a multi-account platform architecture that separates inference from tool execution and gates every agent deployment with CI/CD evaluations. The design addresses the governance gap that appears when you move from a single proof-of-concept agent to a multi-tenant platform serving dozens of workstreams, each with different risk profiles and tool access requirements.

The core idea: split reasoning (inference) from action (tool calls) across account boundaries, route everything through a control plane, and isolate each workstream in its own cell. This prevents a compromised agent in one workstream from reaching tools or data in another, and it gives platform teams a single point to enforce policy, observability, and cost allocation.

Why Separate Inference and Tool Gateways

Most agent frameworks run inference and tool execution in the same process or account. When an LLM decides to call a tool, the orchestrator invokes it directly. This works for demos but creates three problems at scale:

1. Blast radius: A prompt injection or model hallucination can trigger any tool the agent has credentials for.
2. Audit gaps: You cannot easily log, rate-limit, or block tool calls independently of inference requests.
3. Cost attribution: Inference and tool execution costs land in the same bucket, making it hard to charge back workstreams accurately.

AgentCore splits these concerns across two gateways:

• Inference Gateway: Handles LLM requests, prompt templates, and reasoning loops. Lives in a shared inference account with Bedrock endpoints.
• Tool Gateway: Validates, logs, and executes tool calls. Lives in a per-workstream cell account with scoped IAM policies.

The control plane routes agent requests to the correct cell, and the cell's tool gateway enforces per-workstream policies before invoking any external API or database.

Control Plane Architecture

The control plane is a lightweight routing layer that maps incoming agent requests to the correct workstream cell. It does not execute inference or tools itself. Instead, it:

• Authenticates the caller (user, service, or upstream system).
• Looks up the workstream ID from request metadata or a registry.
• Routes the request to the cell's inference gateway via API Gateway or Application Load Balancer.
• Returns a session token or correlation ID for observability.

The control plane runs in its own account and uses AWS Organizations service control policies (SCPs) to prevent cells from calling each other directly. This enforces the hub-and-spoke topology and prevents lateral movement.

Failure Modes

• Control plane down: All agent requests fail. Mitigation is multi-region deployment with Route 53 health checks.
• Cell unreachable: Only that workstream's agents fail. Other cells continue operating.
• Registry out of sync: Requests route to the wrong cell or fail with 404. Mitigation is eventual consistency with DynamoDB global tables or a strongly consistent registry backed by Aurora.

Per-Workstream Cell Isolation

Each workstream gets its own AWS account containing:

• An inference gateway (API Gateway + Lambda or ECS task) that calls Bedrock in the shared inference account via cross-account IAM roles.
• A tool gateway (Lambda or Step Functions) that validates tool calls against a per-workstream allowlist and invokes tools via scoped IAM policies.
• A state store (DynamoDB or S3) for conversation history, session data, and tool call logs.
• Observability collectors (CloudWatch Logs, X-Ray, or third-party APM) scoped to the workstream.

The cell account's IAM policies restrict which Bedrock models, S3 buckets, and external APIs the agent can reach. If a workstream needs access to Salesforce, only that cell's tool gateway gets Salesforce credentials. If another workstream needs Snowflake, only its cell gets Snowflake credentials.

This design also simplifies cost allocation. Each cell's CloudWatch bill, Bedrock token usage, and Lambda invocations roll up to the workstream's account, so finance teams can charge back accurately without parsing logs.

CI/CD Evaluation Pipeline

AgentCore gates every agent deployment with a CI/CD pipeline that runs evaluations before promoting code to production. The pipeline has three stages:

1. Unit tests: Validate prompt templates, tool schemas, and orchestration logic in isolation.
2. Integration tests: Run the agent against a test environment with mocked tools and check that it produces expected outputs for a suite of test cases.
3. Evaluation tests: Run the agent against real tools in a staging cell and measure accuracy, latency, cost, and safety metrics.

The evaluation stage is the critical gate. It uses a test harness that:

• Loads a dataset of input prompts and expected outcomes.
• Invokes the agent for each test case.
• Compares actual tool calls, outputs, and side effects to expected values.
• Computes pass/fail metrics (accuracy, hallucination rate, policy violations).
• Blocks promotion if any metric falls below a threshold.

Example Evaluation Metrics

The pipeline runs in a dedicated CI/CD account with read-only access to production cells. It cannot modify production state, but it can invoke staging cells to run evaluations.

Tool Gateway Implementation

The tool gateway is a Lambda function or Step Functions state machine that:

1. Receives a tool call request from the inference gateway (tool name, parameters, session ID).
2. Validates the tool name against the workstream's allowlist.
3. Validates parameters against a JSON schema.
4. Checks rate limits and quota (e.g., max 100 Salesforce API calls per hour).
5. Logs the request to CloudWatch and X-Ray.
6. Invokes the tool via AWS SDK, HTTP client, or database driver.
7. Returns the result or error to the inference gateway.

Here's a simplified Python example:

import json
import boto3
from aws_lambda_powertools import Logger, Tracer

logger = Logger()
tracer = Tracer()

TOOL_ALLOWLIST = {
    "salesforce_query": {
        "handler": "invoke_salesforce",
        "schema": {"type": "object", "properties": {"query": {"type": "string"}}}
    }
}

@tracer.capture_method
def invoke_salesforce(params):
    # Call Salesforce API with scoped credentials
    client = boto3.client("secretsmanager")
    secret = client.get_secret_value(SecretId="salesforce-creds")
    creds = json.loads(secret["SecretString"])
    # ... make API call, return result
    return {"status": "success", "data": [...]}

@logger.inject_lambda_context
@tracer.capture_lambda_handler
def handler(event, context):
    tool_name = event["tool_name"]
    params = event["parameters"]

    if tool_name not in TOOL_ALLOWLIST:
        logger.error(f"Unauthorized tool: {tool_name}")
        return {"error": "Tool not allowed"}

    tool_config = TOOL_ALLOWLIST[tool_name]
    # Validate params against schema (omitted for brevity)

    handler_fn = globals()[tool_config["handler"]]
    result = handler_fn(params)

    logger.info(f"Tool call succeeded", extra={"tool": tool_name})
    return result

The tool gateway also enforces rate limits using DynamoDB or ElastiCache to track call counts per session or per hour. If a session exceeds the limit, the gateway returns an error and logs a policy violation.

Observability and Debugging

Each cell emits structured logs to CloudWatch Logs with a consistent schema:

• workstream_id: Identifies the cell.
• session_id: Correlates requests across inference and tool gateways.
• tool_name: Which tool was called.
• latency_ms: How long the tool call took.
• error: Any error message or exception.

The control plane aggregates these logs into a central CloudWatch Logs Insights query or ships them to a SIEM like Splunk or Datadog. Platform teams can query across all cells to find slow tools, high error rates, or policy violations.

X-Ray tracing links inference requests to tool calls, so you can visualize the full agent execution path and identify bottlenecks. If an agent takes 10 seconds to respond, X-Ray shows whether the delay is in Bedrock inference, tool execution, or network hops.

Deployment Shape

A typical AgentCore deployment for a mid-sized enterprise looks like this:

• 1 control plane account: Runs the routing layer, registry, and shared observability dashboards.
• 1 shared inference account: Hosts Bedrock endpoints, prompt templates, and inference gateway logic.
• N workstream cell accounts: One per business unit, product team, or risk tier. Each cell is identical except for IAM policies and tool allowlists.
• 1 CI/CD account: Runs the evaluation pipeline, test harnesses, and deployment automation.

The control plane and shared inference account are managed by the platform team. Workstream cells are provisioned via Terraform or CloudFormation StackSets, so adding a new workstream takes minutes. The CI/CD account is locked down with SCPs that prevent it from modifying production state.

When to Use This Architecture

AgentCore's multi-account design makes sense when:

• You have multiple teams or business units deploying agents with different risk profiles.
• You need hard isolation between workstreams for compliance, security, or cost allocation.
• You want to enforce governance (allowlists, rate limits, evaluations) without slowing down individual teams.
• You expect to scale to dozens or hundreds of agents and need a repeatable deployment pattern.

It does not make sense when:

• You have a single agent or a small team experimenting with agentic AI. The overhead of multiple accounts and a control plane is not worth it.
• Your agents do not call external tools or APIs. If all reasoning happens in-context, you do not need a tool gateway.
• You cannot tolerate the latency of cross-account API calls. The control plane and tool gateway add 50-200ms per request.

Technical Verdict

AWS AgentCore is a production-grade blueprint for governed agent platforms. The separation of inference from tool execution, per-workstream cell isolation, and CI/CD evaluation gates address the three hardest problems in multi-tenant agent deployments: blast radius, auditability, and cost attribution.

The architecture is overkill for single-agent experiments but essential for enterprises moving from pilots to production. The open-source reference implementation gives you a working starting point, but you will need to customize IAM policies, tool schemas, and evaluation metrics for your workstreams.

The main trade-off is operational complexity. You are managing N+3 AWS accounts (N workstreams plus control plane, inference, and CI/CD). If your team is not already comfortable with AWS Organizations, cross-account IAM, and infrastructure-as-code, the learning curve is steep. But if you are already running multi-account AWS environments, AgentCore slots in cleanly.

Use it when you need to scale agents across teams with different trust boundaries. Skip it if you are still figuring out whether agents work for your use case.

Source Links

• Building an Agentic AI Platform on Amazon Bedrock AgentCore