Coder Legion - Recent posts tagged bedrock

A Field Guide to Beating Amazon Bedrock Throttling

Tue, 20 Oct 2026 20:53:15 +0000

🛡️ A Field Guide to Beating Amazon Bedrock Throttling

TL;DR

Know your quota first: check Service Quotas, learn which limits are adjustable (soft) and which are not (hard).

Retry with exponential backoff and full jitter on throttling errors. This is the floor, not the ceiling.

Cap concurrency so you stop generating load the quota cannot absorb.

Add a circuit breaker so a throttle storm backs off instead of hammering.

Handle timeouts separately: shrink the batch for model timeouts, tune and retry for client read timeouts.

Use cross-region inference profiles to spread token demand across regions.

Reserve provisioned throughput only when on-demand plus the above is not enough.

If you run real LLM traffic through Amazon Bedrock, throttling is not an edge case. It is the normal operating condition at scale. A ThrottlingException or TooManyRequestsException is Bedrock telling you that your request rate or token rate crossed a quota for that model in that region. The fix is rarely one setting. It is a layered defense: understand the quota, retry intelligently, cap concurrency, degrade gracefully, handle timeouts, and know what to escalate to AWS.

This is the set of strategies I run across production LLM workloads, written as a generic guide. No workload specifics, just the patterns and when each one earns its place.

Step one: know the quota you are hitting

Before writing a line of retry code, find the actual limit. Bedrock quotas are per model, per region, and split by type: tokens per minute (TPM), requests per minute (RPM), and for some models separate input/output token quotas.

Quotas for Amazon Bedrock — the overview, including that default quotas vary by account age, usage history, and region.
Quotas for the bedrock-runtime endpoint — the runtime invoke limits you hit at inference time. View them in the Service Quotas console under Amazon Bedrock.
Scaling and throughput best practices — how per-model TPM and RPM quotas interact.

Soft limits vs hard limits

This distinction decides your whole strategy.

Soft limits (adjustable). In the Amazon Bedrock endpoints and quotas table these are marked Adjustable: Yes. You can raise them. Request an increase through Service Quotas; smaller increases often auto-approve, larger ones route to AWS Support for review.
Hard limits (not adjustable). Marked Adjustable: No. No amount of asking moves these. Your only lever is architectural: reduce demand, spread it across regions, or change the access pattern.

The practical rule: if the limit is soft and you are consistently near it, file the increase and also build the defenses below, because the increase takes time and never makes throttling impossible. If the limit is hard, skip the request entirely and go straight to architecture.

One often-missed detail: Bedrock reserves TPM quota at the start of a request based on input tokens plus maxTokens. If you leave maxTokens unset, it can default to the model maximum and reserve far more quota than your call actually needs, so you throttle earlier than your real token use suggests. Set maxTokens to a realistic ceiling.

Layer one: retry with backoff and jitter

The baseline defense. Catch the throttling error codes (ThrottlingException, TooManyRequestsException), wait, and try again. Two details separate a good retry from a harmful one:

Exponential backoff. Each attempt waits longer: base * 2^attempt, capped at a ceiling. This gives the quota window time to refill.
Full jitter. Randomize the delay inside that window. Without jitter, every throttled caller retries at the same instant and you get a synchronized stampede that throttles again. With jitter, retries spread out.

A compact pattern:

import random, time
from botocore.exceptions import ClientError

THROTTLE = {"ThrottlingException", "TooManyRequestsException"}

def call_with_backoff(fn, *, base=2.0, cap=15.0, max_retries=5):
    for attempt in range(max_retries + 1):
        try:
            return fn()
        except ClientError as exc:
            code = exc.response.get("Error", {}).get("Code", "")
            if code not in THROTTLE or attempt == max_retries:
                raise
            delay = min(base * 2 ** attempt, cap)
            time.sleep(random.uniform(0, delay))  # full jitter

A note on the boto3 client's own retries. Boto3 has a built-in retry mode (standard or adaptive) you set via botocore.config.Config(retries=...). You can lean on it, or disable it (max_attempts=0) and own the retry loop yourself. Owning it is worth the effort when you want the backoff, jitter, and error classification to match the rest of your system rather than two retry layers fighting each other.

The same retry in a Step Functions workflow

If your Bedrock calls run inside a Step Functions state machine, you do not write the backoff loop by hand. You declare it on the task's Retry block and the service applies exponential backoff with jitter for you. Step Functions error handling added a JitterStrategy field so wait times are spread instead of synchronized, the same reason full jitter matters in code.

"InvokeBedrock": {
  "Type": "Task",
  "Resource": "arn:aws:states:::bedrock:invokeModel",
  "Retry": [
    {
      "ErrorEquals": ["ThrottlingException", "TooManyRequestsException"],
      "IntervalSeconds": 2,
      "MaxAttempts": 15,
      "BackoffRate": 2.0,
      "MaxDelaySeconds": 30,
      "JitterStrategy": "FULL"
    }
  ],
  "Catch": [
    { "ErrorEquals": ["States.ALL"], "Next": "HumanReview" }
  ],
  "Next": "NextStep"
}

The fields map onto the code version: IntervalSeconds is the base delay, BackoffRate is the exponential multiplier, MaxDelaySeconds is the cap, MaxAttempts is the retry count, and JitterStrategy: FULL randomizes each wait. The Catch routes to a fallback once retries are exhausted, for example a human-review step, so an exhausted retry degrades gracefully instead of failing the whole execution.

Layer two: cap concurrency

Retries handle the request that already failed. Concurrency caps stop you from generating more load than the quota can absorb in the first place. If 50 workers all call the same model at once, you will throttle no matter how clever the backoff is.

Put a ceiling on in-flight Bedrock calls. In async Python a semaphore does it:

import asyncio

semaphore = asyncio.Semaphore(10)  # max concurrent Bedrock calls

async def bounded_call(fn):
    async with semaphore:
        return await fn()

Upstream of the application, cap concurrency at the infrastructure layer too. If documents arrive as events, put them on an SQS queue and let the SQS-to-Lambda event source mapping enforce a maximum_concurrency. Work waits in the queue until a slot frees, with no polling and no custom counters:

resource "aws_lambda_event_source_mapping" "sqs_to_worker" {
  event_source_arn = aws_sqs_queue.work.arn
  function_name    = aws_lambda_function.worker.arn
  batch_size       = 1
  scaling_config {
    maximum_concurrency = 5
  }
}

Match the concurrency ceiling to the quota, not to how many workers you could theoretically run. The right number is the one that keeps steady-state token rate under the TPM limit.

Layer three: a circuit breaker for throttle storms

Backoff and concurrency caps handle normal pressure. A sustained throttle storm needs something that steps back harder. A circuit breaker watches the throttle rate and, once it crosses a threshold inside a time window, pauses all calls for a cooldown, then resumes at reduced concurrency before restoring full throughput.

The state machine:

Closed (normal): calls flow, throttles are counted.
Open (tripped): after N throttles in the window, pause all calls for a cooldown and halve the concurrency limit.
Half-open (recovering): after the cooldown, resume at the reduced limit; if clean for a while, restore full concurrency; if it throttles again, reopen.

This converts a feedback loop that makes throttling worse (retry, throttle, retry, throttle) into one that actively sheds load until Bedrock recovers. It pairs with the concurrency limiter: the breaker is what turns the limit down and back up.

Layer four: handle timeouts, they are not throttling

Throttling rarely travels alone, and the loudest neighbor is the timeout. There are two distinct kinds, and they want opposite fixes.

Model timeout: shrink the work

A ModelTimeoutException means the request itself was too large to finish in the model's time budget. It is not a rate problem, so retrying the same payload will time it out again. The fix is to reduce the work: halve the batch or chunk the input, then retry smaller.

async def call_shrinking(fn, items, *, max_retries=3):
    batch = list(items)
    for attempt in range(max_retries + 1):
        try:
            return await fn(batch)
        except Exception as exc:
            code = getattr(exc, "response", {}).get("Error", {}).get("Code", "")
            if code != "ModelTimeoutException" or attempt == max_retries:
                raise
            batch = batch[: max(1, len(batch) // 2)]  # halve and retry

Client timeout: tune the budget, then retry the transient ones

A client-side ReadTimeoutError or ConnectTimeoutError is different again: the SDK gave up waiting before the model answered. Large prompts and long generations routinely exceed the boto3 default read timeout, which produces timeouts that look like failures but are really an impatient client.

Two moves:

Set a realistic read_timeout on the boto3 client so a slow-but-valid response is not cut off.
Treat the transient connection timeouts as retryable, alongside throttling, rather than letting them bubble up as hard failures.

from botocore.config import Config

client = boto3.client(
    "bedrock-runtime",
    config=Config(read_timeout=300, retries={"max_attempts": 0}),
)

RETRYABLE = {"ThrottlingException", "TooManyRequestsException"}
# also retry ReadTimeoutError / ConnectTimeoutError by exception class name

Classify before reacting. A throttle wants backoff, a model timeout wants a smaller batch, a client timeout wants a longer budget plus a retry, and a ValidationException wants no retry at all.

Layer five: spread demand with cross-region inference profiles

Quotas are per region. A cross-region inference profile lets a single on-demand call be served from one of several regions, so your token demand is spread across each region's quota instead of piling onto one. For on-demand workloads this is often the highest-leverage change: no capacity to buy, and the effective ceiling goes up because you are drawing from multiple regional pools.

It is not infinite headroom, and it does not change a hard per-model limit in a single region, but for spiky on-demand traffic it buys real room.

Layer six: provisioned throughput, only when you need it

If on-demand plus everything above still throttles under predictable heavy load, Bedrock offers Provisioned Throughput: reserved capacity (model units) with a guaranteed token rate, bought by commitment. It removes throttling for that reserved slice because the capacity is yours.

Treat it as the last layer, not the first. It costs whether or not you use it, so it only makes sense for steady, predictable volume. For bursty or unpredictable traffic, retries, concurrency caps, and cross-region profiles on on-demand are usually the better economics.

Tuning in practice

The layers above are the toolbox. How hard you turn each knob depends on the workload, and a couple of patterns are worth stating plainly.

Match the aggressiveness of retries to the cost of a lost call. Heavy, expensive calls justify more retry attempts, a higher backoff ceiling, and tighter concurrency, because losing one is costly. High-volume, cheaper work is better served by an upstream queue that caps concurrency and a workflow-level self-retry that ends in a manual-review fallback, keeping the per-call retries light. There is no single correct number; the right setting is the one that keeps steady-state token rate under the quota for that workload.

Two caveats that are easy to get wrong:

A "fallback model" can quietly resolve to the same model. If both the primary and fallback point at the same model id through shared config, the fallback functions as extra retries against one model, not as model diversity. Calling it a fallback overstates what it does until the config points at a genuinely different model.
Retry, backoff, and concurrency are not the same as reserved capacity. A defense built entirely from retries, backoff, concurrency caps, and on-demand cross-region inference profiles has no reserved-capacity floor under a true spike. That can be a deliberate on-demand posture, but name it as one, because on-demand plus retries still has a ceiling.

How to decide what to reach for

A short decision order when you see throttling:

Check the quota. Is the limit soft or hard? If soft and you are near it, file the increase now, in parallel with the rest.
Confirm retries have backoff and jitter. If not, fix that first. It is the cheapest win.
Cap concurrency to keep steady-state rate under the TPM limit, at both the app and the infrastructure layer.
Add a circuit breaker if throttle storms are recurring rather than occasional.
Separate timeouts from throttles. Shrink the batch for model timeouts, tune the read timeout and retry for client timeouts.
Spread across regions with a cross-region inference profile for on-demand traffic.
Buy provisioned throughput only for steady, predictable, high volume the above cannot cover.

Throttling is not a wall you hit once and clear. It is a pressure you manage continuously. The systems that stay up are the ones that treat the quota as a first-class design constraint, not an exception to catch.

Your LLM Keeps Making the Same Extraction Mistake. Here's How to Make It Learn.

Sat, 10 Oct 2026 16:09:42 +0000

TL;DR — I open-sourced an AWS sample that makes LLM document extraction learn from human corrections. Each QA fix is captured once and reused, so the pipeline gets more accurate and cheaper over time — with no retraining and no redeployment. Repo: github.com/aws-samples/sample-prompt-correction-memory. You can run the whole self-healing loop locally in 30 seconds, no AWS account needed.

The problem: corrections that evaporate

If you've built an LLM extraction pipeline, you know this pain.

Your QA analyst opens today's invoice. The model extracted payment_terms as "quarterly." That's wrong — "quarterly" is the billing frequency; the actual payment term is Net 30. The analyst corrects it. Done.

Tomorrow, an identical invoice from the same vendor arrives. The model extracts... "quarterly" again. Same mistake. The analyst corrects it again.

The correction evaporated. It fixed one document and taught the system nothing.

Multiply that across thousands of documents and dozens of recurring error patterns, and you get the two options most teams settle for:

Pay analysts to re-fix the same patterns forever (QA cost never goes down), or
Fine-tune a model (expensive, slow, needs ML ops, and you redo it every time the patterns shift).

There's a third way that needs neither.

The idea: treat corrections as durable memory

What if every human correction became a permanent, reusable asset?

That's the pattern behind sample-prompt-correction-memory. Instead of throwing corrections away, it stores them and feeds them back into future extractions — so the same mistake is never made twice.

It works as three tiers, and the magic is that traffic gradually shifts from the expensive tier to the free one:

🟢 Tier 1 — Graduated rules ($0, no LLM call)

When the same correction pattern recurs enough times, the system synthesizes a deterministic rule (a regex, a lookup, a normalization) that handles that field with zero LLM cost and sub-millisecond latency. "Net 30" → 30 becomes a rule; it never needs a model again.

🔵 Tier 2 — Cloud LLM extraction (confidence-gated)

Fields with no matching rule go to Amazon Bedrock (Claude Haiku 4.5). Every field comes back with a confidence score. If confidence clears the threshold, accept it and move on.

🟠 Tier 3 — Self-healing (only when needed)

If confidence is below threshold, the system retrieves the most relevant past corrections, formats them as few-shot examples, and re-extracts with a stronger model (Claude Sonnet 4.5). The model learns from the specific mistakes it made before — in-context, no training.

Every QA correction flows into a correction log (DynamoDB) that powers both the self-healing retrieval and the rule graduation. So the more the system is used, the smarter and cheaper it gets.

Why this matters (the practical payoff)

No ML ops. No retraining, no GPUs, no deployment cycles. A correction takes effect on the very next document.
Cost drops as quality rises. Normally you trade one for the other. Here, recurring fixes graduate to $0 rules — so accuracy climbs while per-document cost falls.
QA effort compounds instead of repeating. Fix it once → fixed forever.
It's a QA action, not an ML project. Improving the pipeline no longer requires the ML team. An analyst uploads a correction; the system improves.

The clearest fit is high-repetition, back-office document processing — accounts payable, contracts, compliance filings, supply-chain docs — where the same field errors recur and analyst time is the real expense.

The architecture

Fully serverless, deployed with AWS SAM:

Document ──► S3 ──► EventBridge ──► SQS ──► Lambda (extract)
                                              │
                          ┌───────────────────┼───────────────────┐
                          ▼                    ▼                   ▼
                   Tier 1: Rules       Tier 2: Bedrock      Tier 3: Self-heal
                   (deterministic)     (Claude Haiku 4.5)   (Claude Sonnet 4.5)
                          ▲                                        │
                          │                                        ▼
                          └──────── Correction Log (DynamoDB) ◄────┘
                                    (grows from QA feedback)

QA corrections are uploaded as JSON to an S3 corrections/ prefix; a second Lambda validates and ingests them into the correction log. Everything is encrypted with a customer-managed KMS key, and IAM is scoped to the specific Bedrock model ARNs the sample uses.

See it in 30 seconds (no AWS account)

The quickstart runs the entire self-healing loop with a mocked Bedrock client, so you can watch the mechanism without deploying anything:

git clone https://github.com/aws-samples/sample-prompt-correction-memory
cd sample-prompt-correction-memory
pip install -e ".[dev]"
python examples/quickstart.py

You'll see output like this:

Step 1: Initial extraction (no correction memory)
  Field:      effective_date
  Value:      March 2024        Confidence: 0.55   ❌ NO
Step 2: Self-healing triggers (0.55 < threshold 0.70)
  Retrieving corrections for 'effective_date'...
  Found: "March 2024" → "2024-03-01"
Step 3: Re-extraction with correction memory
  Value:       2024-03-01       Confidence: 0.95   ✓ YES
  Self-Healed: True

Low-confidence extraction → retrieve past correction → re-extract → correct answer. No retraining. No redeployment.

Deploying to AWS

With the AWS CLI, SAM CLI, and Bedrock model access for Claude Haiku 4.5 + Sonnet 4.5:

make deploy    # S3, DynamoDB, Lambda, EventBridge, SQS, KMS
make seed      # load sample corrections into the correction log
make trigger   # upload a sample document → triggers real extraction
make verify    # read back results (fields, confidence, self_healed)
make destroy   # empty buckets + delete the stack

make verify prints each extracted field with its confidence and whether self-healing kicked in — your proof it works end-to-end.

A companion: spatial memory

This sample provides semantic correction memory (what was extracted wrong, and why). Its companion, sample-textract-field-memory, provides spatial memory (where fields appear on a document layout). Together they form a dual memory for document pipelines: use the cheap spatial lookup when you're confident where a field is, and fall back to self-healing extraction when you're not.

Honest limitations

Because it's more useful when it's used well:

It's sample/reference code, not a production library. Harden IAM, add monitoring, and put a human in the loop for high-stakes fields before real use.
It helps most where the same field errors recur. For wildly heterogeneous, one-off documents, there's less pattern to learn.
Graduated rules currently take effect without a human approval step — in production you'd add a review gate before a synthesized rule starts answering on its own.
Don't use it with regulated data (PII/PHI/PCI) as-is; that path needs a proper compliance review.

Try it and tell me

The repo is open source under aws-samples, MIT-0 licensed, with a benchmark suite, an interactive dashboard, and an offline test harness.

👉 github.com/aws-samples/sample-prompt-correction-memory

What repetitive extraction error is your team still fixing by hand? I'd love to hear which patterns you'd want a system like this to learn.

This is an open-source AWS sample intended for demonstration and non-production use.

Your RAG copilot can't count — stop letting it try

Fri, 31 Jul 2026 11:05:28 +0000

Your RAG copilot can't count — stop letting it try

A user asked our document-search copilot a completely reasonable question: how many documents a specific person authored. The copilot answered with a number. Confidently. With a nice sentence around it.

The real answer, computed by the database, was 84. The copilot said — because it did what every RAG system does when you ask it an aggregation question: it counted what it could see.

Here's the claim I want to defend, because I think a lot of us are shipping this bug right now:

💡 An LLM should never compute an aggregate. Not counts, not sums, not "most recent". If the question is arithmetic over the corpus, the answer comes from the system of record — the model's only job is to phrase it.

🔢 Why the model's count was doomed before it started

This is a multi-agent document-search copilot for a quality-management product, built on LangGraph and Bedrock. When a metadata question comes in ("documents authored by X"), a search agent asks the documents service for matching rows, reranks and permission-gates them, collapses multiple versions of the same document into one card, and surfaces at most 30 results.

Every one of those steps is correct for search. Every one of them destroys the count:

Stage	What it does	What "how many" means here
Documents service (SQL)	runs the filter against the database	true total: 84
Retrieval	returns the top-k rows	capped
View-permission gate	drops rows this user can't see	fewer
Version collapse	5 versions of one doc → 1 card	fewer still
Surfaced set	at most 30 cards	≤ 30, always

The model sits at the bottom of that funnel. When it "counts", it counts the last row of the table. The user asked about the first row. Everything in between is retrieval hygiene that silently turned a database question into a context-window question.

And this is the part that makes it worse than a crash: the answer is plausible. A count that's capped at 30 and deduped looks like a real number. Nobody squints at "you have 27 documents" the way they'd squint at a stack trace. In a quality-management context, someone might put that number in a report.

🪤 The twist: we had the right number the whole time — and overwrote it

This is the bit that stung. The documents service returns the true total with every filtered query. The search node even stored it in agent state, in a field called total_available_results.

Then, a few nodes downstream, the permission-gate node overwrote that same field with the surfaced count — deliberately, and for a good reason:

return {
    "sources": diversified,
    # Honest "documents found": distinct logical documents after the view gate and
    # version-collapse, so the headline count matches the cards (5 versions → 4 docs),
    # not the service's version-row total.
    "total_available_results": len(diversified_all),
    ...
}

That comment is right! If the UI says "found 12" and shows 12 cards, the numbers should agree. The bug wasn't either write in isolation — it was one state field carrying two different meanings at two different points in the pipeline. Upstream it meant "matches in the database"; downstream it meant "cards on screen". Whoever read it last got whichever meaning was freshest.

💡 A state field whose meaning mutates mid-pipeline is a bug factory. If a value is authoritative, give it its own field and never let anything downstream touch it.

🔧 The fix: the database counts, the model narrates

So that's exactly what I did — preserved the service's true total in its own state field, threaded it to the two places that answer the user, and left the surfaced count alone for the card list and pager.

First, the state gets an untouchable field:

class DocumentsSearchState(OrchestratorState, total=False):
    ...
    total_available_results: int   # surfaced: view-gated, version-collapsed, capped at 30

    # The documents service's true match total for the filter, before the 30-cap and
    # version-collapse. Authoritative for a "how many" count; 0 on a content-only
    # search, which has no exact total.
    total_matching_records: int

Then the outcome node — the one that turns search state into both the user-facing status line and the message the agent reasons over — uses the authoritative number in both:

total_matching = state.get("total_matching_records") or 0
headline_total = total_matching if total_matching > total else total

found_msg = (
    f"Showing {len(sources)} of {headline_total} matching documents"
    if headline_total > len(sources)
    else f"Found {len(sources)} matching documents"
)
...
count_note = (
    f" {total_matching} documents match in total "
    "(authoritative count; use this for a count question)."
    if total_matching > len(sources)
    else ""
)
return f"Returned {len(sources)} document cards to the user.{count_note} ..."

And the agent's prompt closes the loop with an explicit instruction:

For a "how many" question, answer with the authoritative total the tool
reports ("N documents match in total"), never the number of cards shown,
which is capped.

Notice the belt-and-braces structure, because it matters: the number goes into the agent's context (so the model can phrase the answer around it) and into the user-facing headline ("Showing 5 of 84 matching documents"). Even on a turn where the model ignores the note and free-styles a count, the UI is showing the true total right next to it. The prompt asks; the headline doesn't have to.

Three small tests pin the behavior: the count note appears when there are more matches than cards, disappears when everything is shown, and never appears for a content-only semantic search — because a similarity ranking has no crisp "matches" set, so pretending it has an exact total would be the same lie in a new costume.

📅 The sibling bug: "the last 10 days" of what?

Same release, same root cause wearing a different hat. Users asked for "documents created in the last 10 days" and the copilot had to turn that into a date filter — without knowing what day it is. A model doesn't know today's date. It will happily guess one from its training data, which is exactly as wrong as counting its context window.

The fix is a tiny LangGraph middleware that injects the current date into the system message on every model call:

def current_date_context(now: datetime | None = None) -> str:
    day = (now or datetime.now(UTC)).date()
    return (
        f"Today's date is {day.isoformat()} (UTC). Resolve any relative date "
        '("today", "yesterday", "the last 10 days", "since March") '
        "to concrete ISO-8601 dates relative to it before building a date filter."
    )


class CurrentDateMiddleware(AgentMiddleware):
    async def awrap_model_call(self, request, handler):
        base = request.system_message
        line = current_date_context()
        merged = line if base is None else f"{base.content}\n\n{line}"
        return await handler(request.override(system_message=SystemMessage(content=merged)))

One non-obvious detail: it's injected per call and never appended to the persisted conversation. Write "today is 2026-07-30" into the thread history and a conversation resumed next week carries a stale date the model will trust completely.

Both fixes are the same principle: the model knows nothing you don't hand it, and it will answer anyway. Counts, dates, totals — anything that's a fact about your system rather than about language has to arrive as context, computed by something that can't hallucinate.

🤔 Where I'd push back on myself

Honest limits of what shipped:

Nothing enforces that the model uses the number. The prompt instructs, the tool message supplies, the headline backstops — but there's no output check asserting the count in the model's prose equals the authoritative one. That's an eval I'd like to have and don't yet.
Content-only searches still have no real count. A semantic ranking over chunks genuinely doesn't have an exact match total, so those turns fall back to the surfaced count. Arguably the honest move there is to refuse to give a number at all. I didn't go that far.
This only covers counts. Sums, averages, "which author has the most" — any of those would need real aggregation endpoints on the service side. The pattern extends; the implementation doesn't yet.

And the design choice I keep turning over: I handed the model the number inline, as prose in a tool result. The agentic-purist alternative is a dedicated count_documents tool the model calls when it detects an aggregation question — cleaner separation, one more round trip, and a new failure mode where the model doesn't call it. I went with inline because the search already paid for the count; a tool call felt like latency for ideology.

So: where do you draw that line — hand the model the number, or hand it a tool to fetch the number? If your copilot answers "how many" questions today, I'd genuinely like to know which side you picked and whether it's held up.

Thanks for making it to the end 🙏 If your assistant has ever announced a total that made you go "wait, that can't be right", I'd love to swap notes on how you fixed it — come find me on LinkedIn.

Building a multi-agent document-search copilot - Part 2: adaptive Hybrid, post-rank permission gate

Tue, 14 Jul 2026 13:19:33 +0000

Building a multi-agent document-search copilot — Part 2: adaptive Hybrid, and a permission gate after the rank

This is Part 2 of two. Part 1 opened on a document-search copilot whose v1 produced muddy results — two retrieval lanes fused into one rerank, where metadata rows that carry no text corrupted the relevance scores. The fix was two reframes: collapse the router into one structured Bedrock call (with a deterministic fallback), and pick exactly one retrieval strategy per query — MetadataOnly, ContentOnly, or Hybrid — instead of fusing the lanes. We left off at the one strategy that refuses to be tidy: Hybrid, where the user wants a topic and filters at the same time.

Picking up where Part 1 stopped, we're now in the back half of the same pipeline — the direct_search → rerank → permission_filter → finalize_results tail of the search graph. Two decisions left: how Hybrid actually retrieves, and where permissions get enforced.

3️⃣ Adaptive Hybrid: peek once, then pick filter-first or rank-then-filter

Hybrid is the interesting one, because "topic + filters" is genuinely a tradeoff and not a single right answer. There was a real debate about it — the kind that goes a few rounds on a whiteboard: run the filter and the rank in parallel (fast, but you have to reconcile two sets), or in sequence (precise, but slower)? We resolved it by refusing to pick — and routing on selectivity instead.

The trick is one cheap question asked once: how many documents match the filters? direct_search peeks the filtered-universe size from the Documents service a single time, with a threshold (default 1000), and branches:

elif strategy == "Hybrid":
    threshold = cfg.get("filter_first_threshold", 1000)
    id_page = cfg.get("filter_id_page_size", 100)
    scope_ids, total_records = await _fetch_filtered_ids(metadata_filters, threshold, id_page)
    if scope_ids is not None:
        candidates = await _content_hits(scope_ids=scope_ids)   # filter-first
        filters_enforced_upstream = True
    else:
        candidates = await _content_hits()                       # rank-then-filter

Bounded set (≤ threshold) → filter-first. The peek returns the actual id set. The semantic query is scoped to exactly those ids via an OpenSearch terms id-scope (over the version field on chunks, the parent field on attachment chunks). Every filter is enforced authoritatively at the source — nothing off-filter can possibly rank, and nothing on-filter is missed. It sets a flag so the later permission gate knows not to re-filter:
```
# filter-first scoped the topic ranking to the service-filtered id set, so the filters
# are already enforced authoritatively; permission_filter must NOT re-enforce them.
filters_enforced_upstream = True
```
Broad set (> threshold) → rank-then-filter. Scoping a semantic query to thousands of ids would be unbounded and slow, so instead we rank the topic unscoped and drop the off-filter hits afterward, locally, with _row_passes_filters — a faithful mirror of the Documents service's operation semantics across every searchable column (string / number / date / boolean / user_list, with contains / equal / greaterThan / the rest):
```
def _row_passes_filters(row: dict, filters: list[dict]) -> bool:
    """Faithful local mirror of the Documents service filter semantics ...
    any hit whose row fails a hard filter is dropped, fail-closed."""
    return all(_row_passes_one(row, f) for f in filters)
```

The thing that makes this safe rather than a sloppy approximation: when the filtered set is huge, the filter is nearly a no-op anyway (it matches most of the library), so the recall ceiling from ranking-first-then-dropping is harmless. The case where dropping would lose good matches — a selective filter — is exactly the case that takes the filter-first path, where nothing is dropped. The branch picks the strategy where its own weakness doesn't apply. I'll be honest: it took me a beat to trust this — it feels like a cheat until you sit with it — but the two failure modes genuinely cancel out.

	filter-first (selective)	rank-then-filter (broad)
Filtered universe	≤ threshold (default 1000)	> threshold
Semantic query	scoped to the filtered id set	unscoped
Filters enforced	at the Documents service (authoritative)	locally, in `permission_filter`
Recall risk	none — nothing off-filter can rank	harmless — filter matches most docs
Round trips for the peek	one (read the id set)	one (just the count, then bail)

One peek, no extra round trips, and the precision/latency tradeoff is made explicit and data-driven instead of being a coin flip baked into the architecture.

The pattern, generalized: when you can't decide between two retrieval strategies, measure the one property that distinguishes them (here, filter selectivity) with a single cheap probe, and route on it. Each branch is chosen precisely where its failure mode is benign.

4️⃣ Cohere rerank, then a fail-closed view gate — and why "after" is fine

The content path is ranked by Cohere (over the top rerank_candidates, default 30), with the OpenSearch fused score / RRF as the fallback when the reranker is unreachable:

results = await bedrock_client.rerank(query=query_text, documents=documents, model_id=model_id)
# ... on any failure or empty result -> _fallback() sorts by the OpenSearch raw score

Metadata results skip the reranker entirely — there's nothing to score — and keep the Documents service order:

if all(c.get("match_kind") == "metadata" for c in candidates):
    return {"reranked": candidates, "cached_top_30": candidates}

Then comes the part that makes security people lean forward in their chair: the per-document view-permission gate runs after rerank, not before retrieval. Stay with me — it's safe, and the reason it's safe is the whole point. permission_filter takes the reranked content hits, confirms each one is viewable via a single bulk export_by_ids call against the system of record, and drops the ones that aren't — fail-closed, so if the service is unreachable, every content hit is dropped rather than leaked:

rows = await documents_service.export_by_ids(content_ids, jwt)
for row in rows:
    dvid = row.get("document_version_id")
    if dvid:
        viewable.add(dvid)
# ... survivors = metadata rows (trusted) + content hits whose id is in `viewable`

Why is gating after rerank safe? Because there are two different boundaries doing two different jobs:

Tenant isolation is the hard wall, and it's enforced at retrieval. Every OpenSearch query is scoped by member_id, and that id comes from the JWT — never from the request body, never from the index. A user physically cannot retrieve another tenant's vectors. That boundary is upstream and absolute.
The post-rerank gate is the soft boundary: per-document view visibility within your own tenant. Drafts, restricted categories, documents shared with three people. That nuance changes hourly, so it does not live in the index — it's checked live, after retrieval, against the source of record.

Rerank only orders the candidates. It exposes nothing — every candidate it sees already passed the tenant wall, and the set it's ordering is bounded (the reranked top set), so the view gate is one bounded bulk call, not a fan-out. The metadata lane skips the gate entirely because its rows came back already view-filtered by the service's own SQL — re-checking trusted rows would be wasted work. Identity from the token, tenant at retrieval, view-visibility after, fail-closed throughout.

The seam I owned on the frontend: a null relevance score

Here's a small but real consequence of "one strategy per query" that landed squarely on my side of the stack — and nearly slipped past me. Content hits carry a relevance score. Metadata hits don't — a satisfied filter is not a relevance measurement, and showing the OpenSearch fused score (which for an exact field match is ~0) would read as "barely relevant" for what is actually a perfect match. So a metadata hit emits a null score in the SSE payload, and the UI has to render that as "matched your filter," not "0% relevant":

def _shape_source_for_emit(s: dict) -> dict:
    # A metadata match is a satisfied filter, not a relevance measure, so its score stays null
    # (the card shows just the match reason); content matches keep their fused/rerank score.
    score = None if s.get("score_signal") == "metadata" else (
        s.get("aggregate_score") or s.get("score") or 0.0
    )
    ...

It's a tiny detail, but it's the kind of thing that breaks if the frontend assumes "every result has a number." A null score is a first-class state — "this matched a filter, there's no relevance to show" — not a missing value to default to zero. Defaulting it to 0.0 would have quietly buried every exact metadata match at the bottom of the list, sorted below loosely-relevant content — a perfect match dead last, and no error anywhere to tell you why. The contract had to carry the absence on purpose.

The throughline across all four decisions — the two in Part 1 and the two here — is the same instinct: make the system commit to one clear shape per turn, and make the tradeoffs explicit instead of averaging them away. One router call instead of three. One retrieval strategy instead of a fused blend. One selectivity probe instead of a parallel-vs-sequential guess. One bounded permission check after a bounded rerank. The muddy-results version tried to do everything at once and let the reranker sort it out. It couldn't. Picking, on purpose, is what made the rank mean something again.

Thanks for sticking through both parts. If you've fought the same muddy-rank fight, I'd genuinely love to hear how you cut yours — the failure modes are weirdly universal. Come say hi on LinkedIn.

Missed the setup? Part 1 covers the muddy-results problem, the single-call router, and the reframe to one strategy per query.

Building a multi-agent document-search copilot — Part 1: muddy results, and one strategy per query

Tue, 14 Jul 2026 13:19:14 +0000

Building a multi-agent document-search copilot — Part 1: muddy results, and one strategy per query

The first version ranked documents badly — and worse, it ranked them badly in a way that looked fine on the architecture diagram. Those are the bugs that get under my skin: every box is green, every arrow points the right way, and the answer is still wrong.

We were building a chat copilot over a regulated document store — the kind where a user types "show me my effective SOPs about equipment cleaning" and expects the right handful of documents back, ranked, with an excerpt and a reason. The v1 design did the obvious thing: run two retrieval lanes in parallel — a structured metadata lane and a semantic content lane — union the hits, rerank the union, render. Clean diagram. Muddy results. We'd open the demo, the pipeline would light up green end to end, and the list that came back was mush: the metadata rows polluted the semantic rank, the relevance scores stopped meaning anything, and there was no clean ordering left to show the user. The architecture was elegant. The experience was not.

This is a two-part story of how that became v2: one strategy per query, never mixed, a router that's a single structured-output call, and a Hybrid path that peeks at the data before it decides how to retrieve. It's an architecture post, so I'll keep it anchored in the specific decisions that actually moved — not a generic "how to build RAG" walkthrough. Part 1 (this post) is the problem and the first two reframes. Part 2 is the hard case — Hybrid — and the permission model.

️ The series at a glance

This is Part 1 of 2.

Part 1 (this post) — the problem and the first two reframes:

️ v1 fused two parallel lanes into one rerank → muddy. Metadata rows have no text; a reranker scores text; fusing them corrupts the one number the UI depends on.
The router is one Bedrock structured-output call, not three sequential hops. Route + rewritten query + strategy + filters come back together, with a deterministic fallback. The catch: the merged task got too hard for a small model, so it runs on a bigger one.
v2 picks exactly one shape per query: MetadataOnly, ContentOnly, or Hybrid. The lanes are never unioned or cross-scored.

Part 2 — the hard case and the safety model:

⚖️ Hybrid is adaptive. It peeks the filtered-universe size once (threshold 1000) and routes on selectivity: small set → scope the semantic query to those ids (filter-first, authoritative); big set → rank unscoped and drop off-filter hits locally (rank-then-filter, harmless recall ceiling).
The view-permission gate runs after rerank, and that's safe — because tenant isolation is enforced earlier, at retrieval, from the token.

The flow, end to end

Here's the whole turn. A message comes in, the supervisor routes it, the search graph runs one retrieval shape, reranks, gates on permissions, and finalizes for the UI:

                 ┌─────────────────────────────────────┐
   user turn ──► │ PLANNER (one Bedrock call)           │
                 │ route + rewrite + strategy + filters │
                 └──────────────┬──────────────────────┘
                                │ deterministic fallback if it fails
                                ▼
         ┌─────────────────── retrieval graph ───────────────────┐
         │   classify_intent                                     │
         │        │  (NoMatch / NeedsClarification skip ahead)   │
         │        ▼                                              │
         │   retrieve        ── picks ONE: Metadata / Content /  │
         │                      Hybrid (adaptive on selectivity) │
         │        ▼                                              │
         │   rank_results    ── Cohere over content; metadata    │
         │                      passes through unscored          │
         │        ▼                                              │
         │   access_gate     ── fail-closed VIEW gate on the     │
         │                      content lane only                │
         │        ▼                                              │
         │   format_response ── floor, shape, SSE                │
         └───────────────────────────────────────────────────────┘

The node order is exactly that, straight from the graph definition:

builder.add_edge(START, "query_rewriting")
builder.add_edge("query_rewriting", "intent_detection")
builder.add_conditional_edges(
    "intent_detection",
    _route_after_intent_detection,
    {"retrieve": "retrieve", "finalize_results": "finalize_results"},
)
builder.add_edge("retrieve", "rerank")
builder.add_edge("rerank", "permission_filter")
builder.add_edge("permission_filter", "finalize_results")

Four decisions made this work. Two of them are in this post; the other two are Part 2. I'll take them in order.

1️⃣ The router is one call, not three

The v1 router ran three sequential Bedrock hops per turn: one to pick a route, one to rewrite the user's query into a clean retrieval string, one to classify intent and extract filters. Three round trips, in series, before any retrieval even started. Each one waits on the last — and the user just watches a spinner the whole time.

My pushback in review was simple: don't run sequential model calls when one call can return the whole decision. A router's job is to emit a structured plan. There's no reason route, rewrite, and intent need to be three separate inferences — they're three fields of one object. So we collapsed them. The supervisor now makes a single structured-output call that returns a typed plan, which gets adapted into the downstream routing contract:

result = await bedrock_client.invoke_with_structured_response_with_fallback_async(
    messages=[SystemMessage(content=system), HumanMessage(content=message)],
    response_structure=Plan,
    chat_model_chain=model_chain,
    max_tokens=1024,
    temperature=0.0,
)

One call, temperature=0.0, and everything the downstream graph needs comes back together: the route, the rewritten query, the strategy, the structured filters. The RouterDecision it adapts into is the contract the search graph consumes:

class RouterDecision(BaseModel):
    route: Literal["documents.search", "documents.doc_context", "general.help"]
    rewritten_query: str = Field(default="")
    strategy: SearchStrategy  # MetadataOnly | ContentOnly | Hybrid | NoMatch | NeedsClarification
    search_value: str = Field(default="")
    filters: list[DocumentFilter] = Field(default_factory=list)
    ...

The twist that bit back. Merging three easy classifications into one harder one means the model now has to do all of it in a single pass — and the small/cheap model we wanted started getting it wrong. So the router moved up to a stronger (and slower) model. The "3 calls → 1 call" math promised a big latency win; the model upgrade promptly taxed a chunk of it back. (Presenting a "latency win" that your own model bump immediately claws into is a humbling little moment — I recommend the experience to no one.) We shipped anyway, because correctness moved the right way and the architecture got dramatically simpler — and because of the second non-negotiable below.

The reusable lesson: a call-merge latency win can be partly clawed back by the accuracy upgrade the merge forces. Budget for that. And always keep a deterministic fallback.

The deterministic fallback is the part I'd flag for anyone copying this. The structured call returns None on any failure (the model is down, the output won't parse, the plan isn't exactly one step), and the caller drops to a non-LLM router:

fallback = route_request(body, intent_resolver=resolve_doc_intent)
plan, router_usage = await supervise_turn(...)
routing = plan_to_router_decision(plan)
# routing is None -> use fallback

The copilot never hangs on a flaky router call. If the smart path can't answer, a dumb-but-reliable path does. That's what lets you run the router on a heavier model without making it a single point of failure.

2️⃣ One strategy per query — the reframe that fixed the muddy results

This is the heart of the v2 change, and the part I argued hardest for — loudly, in more than one meeting.

The v1 retrieval ran two member-scoped lanes in parallel — an OpenSearch hybrid lane (vector + BM25) and a structured metadata lane — and fused them into one set before reranking. The intuition was "more recall is better, let the reranker sort it out." It doesn't work, and the reason is specific: a metadata hit and a content hit are not the same kind of object.

A cross-encoder reranker scores text. You hand it a query and a list of passages, it returns a relevance number per passage. A content chunk has text. A metadata row — "status = effective, author = me" — has no passage. When you stuff that row into the reranker by stringifying a title or a key-id, you get back a number that means nothing, on the same 0-1 scale as the real text scores. Nothing downstream can tell the calibrated score from the garbage one. The rank looks plausible and is quietly wrong.

So v2 picks exactly one retrieval shape per query and runs only that. The strategy comes from the router as a literal:

SearchStrategy = Literal[
    "MetadataOnly", "ContentOnly", "Hybrid", "NoMatch", "NeedsClarification"
]

Strategy	What it means	How it's retrieved	Has a relevance rank?
`MetadataOnly`	structured filters only ("my effective SOPs")	Documents service, filtered rows in service order	No — a satisfied filter isn't a relevance signal
`ContentOnly`	topic search ("policy on cleaning between batches")	OpenSearch hybrid over chunks, unscoped	Yes — Cohere rerank
`Hybrid`	topic and filters ("effective docs about cleaning")	adaptive (Part 2)	Yes — Cohere rerank
`NoMatch`	not a document query (greeting, capability question)	skips retrieval entirely	n/a
`NeedsClarification`	a document query too vague to retrieve ("show me stuff")	skips retrieval, asks a clarifying question	n/a

The lanes are never unioned or cross-scored anymore — retrieve says so in plain terms:

# The lanes are never unioned or cross-scored: a content turn returns content
# candidates, a metadata turn returns metadata candidates.

There's a nice side effect in the graph: NoMatch and NeedsClarification short-circuit straight to finalize_results, skipping retrieval, rerank, and the permission gate. A greeting shouldn't make the UI flash "Searching... / No matches" and shouldn't cost three round trips.

def _route_after_intent_detection(state) -> str:
    return ("finalize_results"
            if state.get("intent_strategy") in ("NoMatch", "NeedsClarification")
            else "retrieve")

The honest cost. Going single-strategy meant deprecating the old global keyword lane entirely. The practical fallout: custom-field values are no longer reachable as a filter, because the content-keyword fold that used to (badly) cover them is gone. That's a real regression on a narrow feature, parked until a dedicated Documents endpoint for custom fields lands. It stings to ship a regression on purpose — but I'd take a clean rank with one honest, documented gap over a muddy rank that hides a dozen.

If you take one thing from this post: don't fuse object types into a single rerank set just to maximize recall. Pick the retrieval shape that matches the query, and run only that. Recall you can't rank cleanly is recall you can't show.

To be continued

So far the story has a tidy shape. One router call instead of three. One retrieval strategy per query instead of a fused blend. Pick MetadataOnly, or ContentOnly, and run only that. Clean.

But I skipped the strategy that refuses to be tidy — the one where the user genuinely wants both at once. "Effective SOPs about equipment cleaning" is a topic and a filter, and you can't honor it by picking a single lane. Running the filter and the rank in parallel is fast, but you have to reconcile two sets. Running them in sequence is precise, but slow. There's no single right answer — which is exactly why it's the interesting part.

That's Hybrid, and in Part 2 a single cheap question turns that coin-flip into a data-driven decision. Then there's the permission gate that runs in a spot that makes security people flinch on first read — after the rank — and why it's actually safe. Finally, the one place all of this landed on my side of the stack: a null relevance score that the frontend has to treat as a first-class state, not a missing number.

Part 2 is live: adaptive Hybrid, and the permission gate that runs after the rank.

Thanks for reading all the way through 🙌 If you've hit the same muddy-rank problem — a pipeline that's green end to end and still returns mush — I'd love to compare notes on how you untangled yours. Come say hi on LinkedIn.

Building a Conversational Chatbot with AWS Bedrock (Amazon Titan)

Sat, 27 Dec 2025 01:06:17 +0000

Building a Conversational Chatbot with AWS Bedrock (Amazon Titan)

Overview

Large Language Models don’t magically “remember” conversations.
In real-world systems, conversation state must be explicitly managed.

In this project, we build a deterministic, production-style conversational chatbot using:

AWS Bedrock
Amazon Titan Text (amazon.titan-text-express-v1)
Python (boto3)

This project demonstrates how teams can safely integrate foundation models into enterprise workflows without giving up control, observability, or reproducibility.

What Is AWS Bedrock?

Amazon Bedrock is a fully managed service that provides access to multiple
foundation models (FMs) via a single API — without requiring you to manage infrastructure.

With Bedrock, you can:

Invoke models securely using IAM
Choose models from different providers
Keep data within AWS (no model training on your prompts by default)
Integrate generative AI directly into existing AWS architectures

Key Bedrock Characteristics

Serverless (no infrastructure management)
Model-agnostic API
Enterprise-grade security
Pay-per-use pricing
Native AWS integration

Business Use Cases of AWS Bedrock

AWS Bedrock is designed for real business workloads, not just demos.

Common Enterprise Use Cases

Internal chatbots & AI copilots
Document summarization & analysis
Automated reporting & insight generation
Semantic search over internal data
AI-assisted debugging & data quality analysis
Analytics narrative generation
Customer support automation

Why Companies Choose Bedrock

Data never leaves AWS
IAM-controlled access
Works seamlessly with S3, Lambda, Glue, Databricks, Redshift
No lock-in to a single model provider

Amazon Titan Text (`amazon.titan-text-express-v1`)

Amazon Titan Text Express is a fast, cost-efficient text generation model built by AWS.

Key Characteristics

Optimized for low-latency text generation
Ideal for chatbots, summarization, and explanations
Deterministic behavior when temperature is low
Fully managed and secured by AWS

When to Use Titan Text Express

Conversational assistants
Structured responses
Enterprise-safe workloads
Cost-sensitive applications

⚠️ Titan does not manage conversation state — which is why explicit memory handling (as shown in this project) is essential.

Architecture Overview

Important:
The entire conversation history is sent on every request.

Core Design Decisions

1️⃣ Explicit Conversation Memory

Amazon Titan does not track sessions.

We:

Store user and assistant messages
Append them to a history list
Inject the full history into each prompt

This makes the system:

Predictable
Auditable
Easy to debug

2️⃣ Role-Based Prompt Formatting

Conversation is formatted as:

User: ...
Assistant: ...

This significantly improves response quality and consistency.

3️⃣ Stop Sequences

We configure:

"stopSequences": ["User:"]

This prevents the model from hallucinating the next user message.

4️⃣ Deterministic Generation

Low temperature

Explicit assistant cue

Token limits

How to Run the Project

Clone the Github repo AWS Bedrock Chatbot(Titan)
Prerequisites

Python 3.9+
AWS credentials configured

Install Dependencies
pip install boto3

Run the Chatbot
python chatbot.py


Type exit to quit

You can also experiment with other models available on the AWS Bedrock service page.

On the AWS Bedrock service page Click on Model Catalog . Here you have access to other model providers like Meta, Anthropic, Mistral AI etc. You can search for different models from different or same providers.

Click on a Model, read it's documentation to understand how to use it in your project and copy the model's ID

Evaluating Amazon Bedrock Knowledge Base integration

Wed, 18 Jun 2025 12:51:19 +0000

Amazon Bedrock Knowledge Base pricing is based on the following components:

1. Storage and Ingestion

Storage: You are charged for the storage of vector embeddings in the underlying vector database (e.g., Amazon OpenSearch, Pinecone, or Redis Enterprise Cloud).
Ingestion: When you ingest (sync) documents into the knowledge base, you are charged for the number of tokens processed to create embeddings.

Note: The actual storage and ingestion costs depend on the vector database you select. Bedrock itself does not charge a separate fee for the knowledge base feature, but you pay for the underlying vector store and the compute used for embedding generation.

2. Embedding Model Usage

When you ingest documents, Bedrock uses an embedding model to convert your text into vectors.
You are charged for the number of input tokens processed by the embedding model during ingestion.
The price per 1,000 tokens depends on the embedding model you select (e.g., Amazon Titan Embeddings, Cohere Embed, etc.).

3. Vector Database (Vector Store) Costs

You pay for the storage and retrieval operations in the vector database you choose (e.g., OpenSearch, Pinecone, Redis).
These costs are billed separately by the respective service (e.g., OpenSearch Service pricing, Pinecone pricing, etc.).

4. Retrieval (Query) Costs

When you query the knowledge base (e.g., via an agent), you are charged for:
- The number of tokens processed by the embedding model (if your query is embedded).
- The inference cost of the foundation model used to generate the answer (standard Bedrock model inference pricing).

Example Cost Flow

Ingest 10,000 tokens into the knowledge base:
- Charged for 10,000 tokens by the embedding model.
- Charged for storage in the vector database.
Query the knowledge base:
- Charged for embedding the query (if applicable).
- Charged for retrieval from the vector database.
- Charged for the foundation model inference to generate the answer.

Where to See Pricing

Summary Table

Component	How Charged
Embedding Model	Per 1,000 tokens ingested/queried
Vector Database Storage	Per GB/month (varies by provider)
Vector Database Retrieval	Per query (varies by provider)
Model Inference	Per 1,000 tokens generated (standard Bedrock FM)

References:

Coder Legion - Recent posts tagged bedrock

A Field Guide to Beating Amazon Bedrock Throttling

🛡️ A Field Guide to Beating Amazon Bedrock Throttling

Step one: know the quota you are hitting

Soft limits vs hard limits

Layer one: retry with backoff and jitter

The same retry in a Step Functions workflow

Layer two: cap concurrency

Layer three: a circuit breaker for throttle storms

Layer four: handle timeouts, they are not throttling

Model timeout: shrink the work

Client timeout: tune the budget, then retry the transient ones

Layer five: spread demand with cross-region inference profiles

Layer six: provisioned throughput, only when you need it

Tuning in practice

How to decide what to reach for

Your LLM Keeps Making the Same Extraction Mistake. Here's How to Make It Learn.

The problem: corrections that evaporate

The idea: treat corrections as durable memory

🟢 Tier 1 — Graduated rules ($0, no LLM call)

🔵 Tier 2 — Cloud LLM extraction (confidence-gated)

🟠 Tier 3 — Self-healing (only when needed)

Why this matters (the practical payoff)

The architecture

See it in 30 seconds (no AWS account)

Deploying to AWS

A companion: spatial memory

Honest limitations

Try it and tell me

Your RAG copilot can't count — stop letting it try

Your RAG copilot can't count — stop letting it try

🔢 Why the model's count was doomed before it started

🪤 The twist: we had the right number the whole time — and overwrote it

🔧 The fix: the database counts, the model narrates

📅 The sibling bug: "the last 10 days" of what?

🤔 Where I'd push back on myself

Building a multi-agent document-search copilot - Part 2: adaptive Hybrid, post-rank permission gate

Building a multi-agent document-search copilot — Part 2: adaptive Hybrid, and a permission gate after the rank

3️⃣ Adaptive Hybrid: peek once, then pick filter-first or rank-then-filter

4️⃣ Cohere rerank, then a fail-closed view gate — and why "after" is fine

The seam I owned on the frontend: a null relevance score

Building a multi-agent document-search copilot — Part 1: muddy results, and one strategy per query

Building a multi-agent document-search copilot — Part 1: muddy results, and one strategy per query

️ The series at a glance

The flow, end to end

1️⃣ The router is one call, not three

2️⃣ One strategy per query — the reframe that fixed the muddy results

To be continued

Building a Conversational Chatbot with AWS Bedrock (Amazon Titan)

Building a Conversational Chatbot with AWS Bedrock (Amazon Titan)

Overview

What Is AWS Bedrock?

Key Bedrock Characteristics

Business Use Cases of AWS Bedrock

Common Enterprise Use Cases

Why Companies Choose Bedrock

Amazon Titan Text (amazon.titan-text-express-v1)

Key Characteristics

When to Use Titan Text Express

Architecture Overview

Core Design Decisions

1️⃣ Explicit Conversation Memory

2️⃣ Role-Based Prompt Formatting

3️⃣ Stop Sequences

Evaluating Amazon Bedrock Knowledge Base integration

1. Storage and Ingestion

2. Embedding Model Usage

3. Vector Database (Vector Store) Costs

4. Retrieval (Query) Costs

Example Cost Flow

Where to See Pricing

Summary Table

Amazon Titan Text (`amazon.titan-text-express-v1`)