A canary deployment on Google Cloud Run sends a new build to a small slice of real users before everyone gets it. Cloud Run makes this cheap because every deploy already creates an immutable revision, and traffic can be split between revisions with one command. This tutorial walks through 13 steps, about 75 minutes of hands-on time, and ends with a Cloud Build pipeline that ships a shadow revision, tests it, shifts traffic in stages, and rolls back on its own when the numbers go bad.
Containers keep absorbing more workloads. Gartner predicted that more than 75% of AI and machine-learning deployments would run on container technology by 2027, up from fewer than 50% in 2024. More containers means more releases, and more releases means more chances to break production on a Friday. If you already followed our Google Cloud Run setup walkthrough, this guide picks up where it stops: the release process, not the first deploy.
One clarification up front. Google’s October 2026 guidance on continuous evaluation uses the terms live and shadow revisions, but a shadow revision is not a built-in Cloud Run mode that mirrors production traffic. It is a pattern. You deploy with --no-traffic, give the revision a tag, and call it through its tagged URL while it receives 0% of configured traffic. Cloud Build orchestrates the steps, and a plain Cloud Run traffic-update command does the promotion. Nothing in this tutorial depends on a preview feature.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
Why Canary Deployment Fits Cloud Run So Well
Most platforms make you bolt on a service mesh or an ingress controller before you can split traffic by percentage. Cloud Run ships the primitive. Each deployment creates a new revision that holds a container image plus its configuration, and each service has a stable run.app URL that routes to whichever revisions you pick. A revision can receive 0% of production traffic and still answer requests on a tagged revision URL. That one fact unlocks both the shadow test and the percentage-based canary.
Compare that with running the same pattern on Kubernetes. The Google Kubernetes Engine setup guide shows how much machinery a cluster needs before you can even think about weighted routing. For teams weighing managed options, our Fargate vs Cloud Run vs Container Apps comparison covers how the three serverless container platforms handle revisions and traffic.
Before building anything, it helps to see where a canary sits among the common release strategies. The table below compares four of them on the dimensions that matter once real users are involved.
| Strategy | How traffic moves | Blast radius if the release is bad | Rollback speed | Cloud Run fit |
|---|---|---|---|---|
| Big-bang | 100% switches to the new revision at once | All users | Fast, but after the damage | Default behavior of gcloud run deploy |
| Blue-green | Old and new run side by side, then flip to 100% | All users after the flip | Fast (flip back) | Tag the new revision, then promote |
| Canary | Small share first, then stages | Only the canary share | Fast (set share to 0) | Native through revision traffic splits |
| Shadow (0% revision) | No production traffic, only test traffic on a tagged URL | None | Nothing to roll back | Native through --no-traffic and tags |
The strongest setup combines the last two rows. The shadow revision catches crashes, bad configuration and broken health checks without exposing anyone. The canary then catches the problems only real traffic reveals, such as a slow query under production data or a client that sends an odd header. We build both stages here.
What the pipeline will do
- Build the container image with Cloud Build and push it to Artifact Registry.
- Deploy a new revision with
--no-trafficand a unique tag, so production users never see it. - Run a gate script against the tagged URL that checks status codes, error rate and latency.
- Shift 5% of traffic to the revision, watch it, then move to 25%, 50% and 100%.
- Roll back to the previous revision automatically if any stage fails its check.
Cloud Build does the orchestration. It does not split traffic itself. The split is a Cloud Run operation, and Cloud Build just calls the gcloud command at the right moment. Keeping that division clear makes debugging far easier later. Cloud Deploy, Google’s delivery-pipeline service, can also target Cloud Run, but the documented shadow pattern does not require it, so this guide skips it.
Prerequisites and Versions
You need a Google Cloud project with billing enabled, a terminal, and about 75 minutes. Everything below runs on a laptop, in Cloud Shell, or on a CI box. The sample service is a small Node.js API, but the pipeline does not care which language your real service uses. Google’s release notes list patched Node.js runtimes at 22.23.4, 24.22.1 and 26.12.1, so the tutorial targets the Node.js 24 line for the container image.
| Requirement | Version or level | Why you need it |
|---|---|---|
| Google Cloud project | Billing enabled | Cloud Run, Cloud Build and Artifact Registry all bill to a project |
Google Cloud CLI (gcloud) | Latest release (run gcloud components update) | Deploys revisions, tags them and edits traffic |
| Node.js | 24 LTS line | Runs the sample API locally |
| Python | 3.10 or newer | Runs the gate script (standard library only) |
| Docker | Optional | Only for local image tests, because Cloud Build builds remotely |
| IAM role on your user | Owner, or Editor plus Project IAM Admin | Needed once to enable APIs and grant roles |
| Shell | bash or zsh | Commands use bash variable syntax |
Check your tooling before you start. The commands below print the versions you have installed and should return without errors.
gcloud --version
node --version
python3 --version
gcloud auth list
gcloud config get-value projectA note on naming. Google’s documentation now calls Cloud Functions Cloud Run functions, so older tutorials that mention Cloud Functions describe the same product line you use here. If you only need a single function, it deploys into the same revision and traffic system covered in this guide.
Set Up the Project, Registry and Service Accounts
The first three steps prepare the Google Cloud side. They take about 15 minutes, and most of that is waiting for API enablement. Do them in order, because the later steps assume the registry and service accounts exist.
Step 1: Set variables and enable the APIs
Pick a project ID and a region, then export them as shell variables so every later command stays copy-paste safe. The tutorial uses us-central1, but any Cloud Run region works. Enabling the APIs is a one-time cost and does not bill you by itself.
export PROJECT_ID="your-project-id"
export REGION="us-central1"
export SERVICE="ledger-api"
export REPO="canary-demo"
gcloud config set project "$PROJECT_ID"
gcloud config set run/region "$REGION"
gcloud services enable \
run.googleapis.com \
cloudbuild.googleapis.com \
artifactregistry.googleapis.com \
logging.googleapis.com \
monitoring.googleapis.comExpected output is a line reading Operation "operations/..." finished successfully. If the command hangs for more than two minutes, check that billing is attached to the project, because Cloud Build refuses to enable without it.
Step 2: Create an Artifact Registry repository
Cloud Run pulls container images from Artifact Registry. Source-based deploys would create a cloud-run-source-deploy repository for you, but a canary pipeline benefits from a repository you control, with a name you can reference in IAM and cleanup policies. Create a Docker-format repository in the same region as the service to avoid cross-region pull latency and egress charges.
gcloud artifacts repositories create "$REPO" \
--repository-format=docker \
--location="$REGION" \
--description="Canary tutorial images"
gcloud artifacts repositories list --location="$REGION"Note that stored images can incur Artifact Registry storage charges even when the service scales to zero. Google’s documentation points to its pricing calculator for estimates, and Step 13 shows how to delete old images so the bill stays flat.
Step 3: Create two service accounts with least privilege
Use two identities. The runtime account is what your container runs as, and it should hold almost no roles. The builder account is what Cloud Build uses to push images and move traffic, so it needs more. Splitting them means a compromised container cannot rewrite your traffic split.
gcloud iam service-accounts create canary-runtime \
--display-name="Canary demo runtime"
gcloud iam service-accounts create canary-builder \
--display-name="Canary demo builder"
RUNTIME_SA="canary-runtime@${PROJECT_ID}.iam.gserviceaccount.com"
BUILDER_SA="canary-builder@${PROJECT_ID}.iam.gserviceaccount.com"
for ROLE in roles/run.admin roles/artifactregistry.writer roles/logging.logWriter roles/logging.viewer; do
gcloud projects add-iam-policy-binding "$PROJECT_ID" \
--member="serviceAccount:${BUILDER_SA}" --role="$ROLE"
done
gcloud iam service-accounts add-iam-policy-binding "$RUNTIME_SA" \
--member="serviceAccount:${BUILDER_SA}" \
--role="roles/iam.serviceAccountUser"The last binding is the one people forget. Deploying a revision that runs as another service account requires the deployer to hold iam.serviceAccountUser on that account. Without it, Step 11 fails with a permission error that looks unrelated to IAM. The table below lists which identity needs which role.
| Identity | Role | Scope | Reason |
|---|---|---|---|
| canary-builder | roles/run.admin | Project | Deploy revisions, set tags, update traffic |
| canary-builder | roles/artifactregistry.writer | Project | Push built images |
| canary-builder | roles/logging.logWriter | Project | Write build logs when using a custom account |
| canary-builder | roles/logging.viewer | Project | Let the pipeline read error logs for the new revision |
| canary-builder | roles/iam.serviceAccountUser | On canary-runtime only | Allow deploys that run as the runtime account |
| canary-runtime | None by default | Not applicable | The demo app calls no other Google APIs |
Your runtime account may need Artifact Registry Reader if the image lives in a different project. In this tutorial the registry sits in the same project, so Cloud Run can pull the image without extra grants.
Build the Sample Service That Can Fail on Demand
A canary pipeline is only believable if you watch it catch a bad release. The sample service therefore has two knobs controlled by environment variables: FAIL_RATE makes a share of /checkout calls return HTTP 500, and LATENCY_MS adds artificial delay. Version 1 runs clean. Version 2 will ship with a deliberate fault so you can see the gate stop it.
Step 4: Write the application
The app uses only Node’s built-in http module, so there is no npm install step and no dependency to patch. It prints one JSON line per request. Cloud Logging parses the severity and message fields automatically, and Cloud Run injects K_REVISION so every log line names the revision that served it. That field is what lets you separate canary logs from baseline logs later.
// app/server.js
const http = require("node:http");
const PORT = Number(process.env.PORT || 8080);
const VERSION = process.env.APP_VERSION || "dev";
const REVISION = process.env.K_REVISION || "local";
const FAIL_RATE = Number(process.env.FAIL_RATE || 0);
const LATENCY_MS = Number(process.env.LATENCY_MS || 0);
function log(severity, message, extra = {}) {
console.log(JSON.stringify({ severity, message, revision: REVISION, version: VERSION, ...extra }));
}
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
const server = http.createServer(async (req, res) => {
const started = Date.now();
const send = (status, body) => {
res.writeHead(status, { "content-type": "application/json" });
res.end(JSON.stringify(body));
log(status >= 500 ? "ERROR" : "INFO", "request", {
path: req.url, status, ms: Date.now() - started,
});
};
if (req.url === "/healthz") return send(200, { ok: true });
if (req.url === "/version") return send(200, { version: VERSION, revision: REVISION });
if (req.url.startsWith("/checkout")) {
if (LATENCY_MS > 0) await sleep(LATENCY_MS);
if (Math.random() < FAIL_RATE) return send(500, { error: "simulated failure" });
return send(200, { order: "accepted", version: VERSION });
}
return send(404, { error: "not found" });
});
server.listen(PORT, () => log("INFO", "listening", { port: PORT }));
process.on("SIGTERM", () => server.close(() => process.exit(0)));Two details matter for Cloud Run. The server must listen on the port named in the PORT variable, and it should handle SIGTERM so in-flight requests finish when an instance shuts down. Missing either one produces confusing failures during traffic shifts, which the troubleshooting table covers later.
Step 5: Add the Dockerfile and test locally
The Dockerfile uses a slim Node 24 base image and drops to the unprivileged node user. Rebuild the image regularly. Google patched its Node.js, Go, Java, Python and Ruby runtimes in October 2026 and told users to rebuild and redeploy to pick the fixes up, so a pinned image that never rebuilds drifts behind.
# app/Dockerfile
FROM node:24-slim
WORKDIR /app
COPY server.js .
ENV NODE_ENV=production
USER node
EXPOSE 8080
CMD ["node", "server.js"]Run the app on your machine first. Local testing costs nothing and catches typos before they burn a Cloud Build minute.
cd app
APP_VERSION=1.0.0 node server.js &
curl -s localhost:8080/version
curl -s localhost:8080/checkout
kill %1You should see {"version":"1.0.0","revision":"local"} and {"order":"accepted","version":"1.0.0"}. If the second call prints an error instead, check that FAIL_RATE is unset in your shell.
Deploy Version 1 as the Production Baseline
A canary needs something to compare against. This section ships a stable revision that takes 100% of traffic. You build it once by hand here, and from Step 11 onward the pipeline does the same work for you.
Step 6: Build the image and deploy the first revision
Build with Cloud Build so you do not need Docker locally. The image path follows the Artifact Registry format REGION-docker.pkg.dev/PROJECT/REPO/IMAGE:TAG. Tag images with the version number, never with latest, because a canary pipeline must know exactly which bytes each revision runs.
export IMAGE_BASE="${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO}/${SERVICE}"
gcloud builds submit app --tag "${IMAGE_BASE}:1.0.0"
gcloud run deploy "$SERVICE" \
--image "${IMAGE_BASE}:1.0.0" \
--service-account "$RUNTIME_SA" \
--allow-unauthenticated \
--port 8080 \
--max-instances 10 \
--update-env-vars APP_VERSION=1.0.0The deploy command ends with output similar to the block below. The service URL is stable across every future revision, and users never need to know it changed.
Deploying container to Cloud Run service [ledger-api] in project [your-project-id] region [us-central1]
OK Deploying new service... Done.
OK Creating Revision...
OK Routing traffic...
OK Setting IAM Policy...
Done.
Service [ledger-api] revision [ledger-api-00001-abc] has been deployed and is serving 100 percent of traffic.
Service URL: https://ledger-api-xxxxxxxxxx-uc.a.run.appThe --allow-unauthenticated flag keeps the demo simple. Some organizations block it through a domain-restricted sharing policy. If yours does, skip the flag and send an identity token with each request, a technique the Advanced Tips section covers.
Step 7: Confirm the baseline is healthy
Capture the service URL into a variable and hit the endpoints. Then list revisions so you know the exact name of the stable one, because the rollback logic in Step 11 needs it.
export SERVICE_URL=$(gcloud run services describe "$SERVICE" \
--format='value(status.url)')
curl -s "$SERVICE_URL/version"
curl -s "$SERVICE_URL/checkout"
gcloud run revisions list --service "$SERVICE"
gcloud run services describe "$SERVICE" --format='value(status.traffic)'Expect {"version":"1.0.0","revision":"ledger-api-00001-abc"} from the first call and an accepted order from the second. The revisions list shows one active revision. Keep this terminal open, because you will compare against it throughout the next sections.
Ship a Shadow Revision at 0% Traffic
This is the core move of the whole tutorial. You deploy the new version with --no-traffic and a unique tag, so production users never touch it. Cloud Run creates the revision, gives it a tag, and exposes it on a URL of the form https://TAG---SERVICE-HASH-REGION.a.run.app. The triple hyphen separates the tag from the service name, which trips up people who try to guess the address.
Step 8: Build version 2 and deploy it with a tag and no traffic
Rebuild the image as 2.0.0. Then deploy it with --no-traffic and --tag canary. The environment variable FAIL_RATE=0.2 simulates a regression where one checkout in five fails. Use --update-env-vars rather than --set-env-vars, since the second flag wipes every existing variable and a real service usually has several.
gcloud builds submit app --tag "${IMAGE_BASE}:2.0.0"
gcloud run deploy "$SERVICE" \
--image "${IMAGE_BASE}:2.0.0" \
--no-traffic \
--tag canary \
--update-env-vars APP_VERSION=2.0.0,FAIL_RATE=0.2Service [ledger-api] revision [ledger-api-00002-def] has been deployed and is serving 0 percent of traffic.
The revision can be reached directly at https://canary---ledger-api-xxxxxxxxxx-uc.a.run.appThe line “serving 0 percent of traffic” is the confirmation you want. Your stable revision still answers every real request. Tags are also not the same thing as Google Cloud resource labels. A tag creates a routable URL and can be used in traffic commands, while labels only organize resources for billing and filtering.
Step 9: Smoke-test the tagged URL by hand
Hit the canary URL directly and compare it with the production URL. A manual check teaches you what the gate script will automate in the next step.
export CANARY_URL=$(gcloud run services describe "$SERVICE" --format=json \
| python3 -c "import json,sys; t=json.load(sys.stdin)['status']['traffic']; print(next(x['url'] for x in t if x.get('tag')=='canary'))")
echo "$CANARY_URL"
curl -s "$CANARY_URL/version"
for i in $(seq 1 10); do curl -s -o /dev/null -w "%{http_code} " "$CANARY_URL/checkout"; done; echo
for i in $(seq 1 10); do curl -s -o /dev/null -w "%{http_code} " "$SERVICE_URL/checkout"; done; echoThe canary loop will print a mix such as 200 200 500 200 200 200 500 200 200 200, while the production loop prints ten 200s. That contrast is the entire point. You found a regression with zero user impact, and the next step teaches a script to find it for you.
Write the Gate Script That Decides Go or No-Go
A gate turns a human glance into a repeatable decision. The script sends a fixed number of requests to a URL, measures the error rate and the 95th percentile latency, and exits with code 1 if either breaches a threshold. Cloud Build treats a non-zero exit as a failed step and stops the pipeline, so the gate needs no special integration.
Step 10: Create gate.py with the standard library only
Using only the standard library means the script runs in the stock Cloud SDK builder image without any pip install. It sends requests from a small thread pool to produce some concurrency, which surfaces issues like connection limits that a single-threaded loop would miss. An optional TOKEN environment variable adds a bearer header for private services.
# gate/gate.py
import argparse, json, os, sys, time, urllib.request, urllib.error
from concurrent.futures import ThreadPoolExecutor
def hit(url):
req = urllib.request.Request(url)
token = os.environ.get("TOKEN")
if token:
req.add_header("Authorization", f"Bearer {token}")
start = time.perf_counter()
try:
with urllib.request.urlopen(req, timeout=10) as resp:
status = resp.status
except urllib.error.HTTPError as err:
status = err.code
except Exception:
status = 599
return status, (time.perf_counter() - start) * 1000
def main():
p = argparse.ArgumentParser()
p.add_argument("--url", required=True)
p.add_argument("--path", default="/checkout")
p.add_argument("--requests", type=int, default=200)
p.add_argument("--workers", type=int, default=8)
p.add_argument("--max-error-rate", type=float, default=0.01)
p.add_argument("--max-p95-ms", type=float, default=500)
a = p.parse_args()
with ThreadPoolExecutor(max_workers=a.workers) as pool:
results = list(pool.map(hit, [a.url + a.path] * a.requests))
errors = sum(1 for s, _ in results if s >= 500)
lats = sorted(ms for _, ms in results)
p95 = lats[int(len(lats) * 0.95) - 1]
rate = errors / len(results)
summary = {"requests": len(results), "errors": errors,
"error_rate": round(rate, 4), "p95_ms": round(p95, 1)}
print(json.dumps(summary))
if rate > a.max_error_rate or p95 > a.max_p95_ms:
print("GATE FAILED", file=sys.stderr)
sys.exit(1)
print("GATE PASSED")
if __name__ == "__main__":
main()Run it against both URLs to prove it can tell good from bad. The thresholds default to a 1% error budget and a 500 ms p95 ceiling. Tune them to your service’s real service-level objectives, and never copy mine blindly.
python3 gate/gate.py --url "$SERVICE_URL"
python3 gate/gate.py --url "$CANARY_URL"; echo "exit code: $?"{"requests": 200, "errors": 0, "error_rate": 0.0, "p95_ms": 118.4}
GATE PASSED
{"requests": 200, "errors": 41, "error_rate": 0.205, "p95_ms": 131.9}
GATE FAILED
exit code: 1The latency numbers above are illustrative and will differ on your network. The pattern is what counts. The clean revision passes, the faulty one fails with an error rate near the 20% you injected, and the exit code carries that verdict to the pipeline. A sample size of 200 is enough to catch a 20% fault, but a subtle 1% regression needs thousands of requests, which the Advanced Tips section addresses.
Automate the Canary Deployment with Cloud Build
Manual commands prove the idea, and a pipeline makes it routine. The Cloud Build design here follows the pattern in Google’s updated Cloud Run guidance: build the image, push it, deploy a revision without traffic, evaluate it on its tagged URL, and promote only if the evaluation passes. The repository layout is small enough to hold in your head.
canary-demo/
app/
server.js
Dockerfile
gate/
gate.py
ci/
snapshot.sh
canary.sh
cloudbuild.yamlStep 11: Write the helper scripts and cloudbuild.yaml
Start with ci/snapshot.sh. Before the pipeline deploys anything, it records which revision currently holds 100% of traffic. That name is the rollback target, and capturing it up front avoids guessing later when two revisions share the split.
#!/usr/bin/env bash
# ci/snapshot.sh SERVICE REGION
set -euo pipefail
SERVICE="$1"; REGION="$2"
gcloud run services describe "$SERVICE" --region "$REGION" --format=json \
| python3 -c "
import json, sys
traffic = json.load(sys.stdin)['status']['traffic']
stable = [t for t in traffic if t.get('percent') == 100]
if not stable:
sys.exit('No revision holds 100 percent of traffic. Fix the split first.')
print(stable[0]['revisionName'])
" > /workspace/stable.txt
echo "Stable revision: $(cat /workspace/stable.txt)"Next comes ci/canary.sh, the script that does the real work. It looks up the new revision through its tag, runs the gate against the 0% shadow, then walks through traffic stages. At every stage it waits for a soak period, runs the gate again and counts error-level log entries for the new revision. A shell trap restores the stable revision on any failure, so rollback needs no separate pipeline step.
#!/usr/bin/env bash
# ci/canary.sh SERVICE REGION TAG
set -euo pipefail
SERVICE="$1"; REGION="$2"; TAG="$3"
STAGES="${STAGES:-5 25 50}"
SOAK_SECONDS="${SOAK_SECONDS:-60}"
MAX_ERROR_RATE="${MAX_ERROR_RATE:-0.01}"
MAX_P95_MS="${MAX_P95_MS:-500}"
MAX_LOG_ERRORS="${MAX_LOG_ERRORS:-0}"
STABLE="$(cat /workspace/stable.txt)"
read -r NEW CANARY_URL < <(gcloud run services describe "$SERVICE" --region "$REGION" --format=json \
| python3 -c "
import json, sys
tag = '$TAG'
hit = [t for t in json.load(sys.stdin)['status']['traffic'] if t.get('tag') == tag]
print(hit[0]['revisionName'], hit[0]['url'])
")
echo "Stable: $STABLE | Candidate: $NEW | URL: $CANARY_URL"
rollback() {
echo "ROLLBACK: sending 100 percent back to $STABLE" >&2
gcloud run services update-traffic "$SERVICE" --region "$REGION" --to-revisions "$STABLE=100"
}
trap 'rc=$?; if [ "$rc" -ne 0 ]; then rollback; fi' EXIT
gate() {
python3 gate/gate.py --url "$CANARY_URL" \
--max-error-rate "$MAX_ERROR_RATE" --max-p95-ms "$MAX_P95_MS"
}
echo "== Shadow gate (0 percent traffic) =="
gate
for PCT in $STAGES; do
REST=$((100 - PCT))
echo "== Stage: $PCT percent to $NEW =="
gcloud run services update-traffic "$SERVICE" --region "$REGION" \
--to-revisions "$STABLE=$REST,$NEW=$PCT"
sleep "$SOAK_SECONDS"
gate
ERRS=$(gcloud logging read \
"resource.type=\"cloud_run_revision\" AND resource.labels.revision_name=\"$NEW\" AND severity>=ERROR" \
--freshness="${SOAK_SECONDS}s" --format='value(timestamp)' | wc -l)
echo "Error log entries in last ${SOAK_SECONDS}s: $ERRS"
if [ "$ERRS" -gt "$MAX_LOG_ERRORS" ]; then
echo "Too many error logs at $PCT percent" >&2
exit 1
fi
done
echo "== Promote: 100 percent to $NEW =="
gcloud run services update-traffic "$SERVICE" --region "$REGION" --to-revisions "$NEW=100"
echo "Promotion complete."Look closely at the promotion line. It names the revision explicitly instead of using --to-latest. The latest flag pins the service to always send traffic to the newest revision, and once a service is in that mode, every later deploy takes 100% of traffic immediately, even with --no-traffic in your script’s plan. Explicit revision names keep the split under your control.
Now tie everything together in cloudbuild.yaml. It runs as the dedicated builder account from Step 3, logs to Cloud Logging only, and allows 30 minutes, which is generous for three 60-second soaks plus builds.
# cloudbuild.yaml
substitutions:
_SERVICE: ledger-api
_REGION: us-central1
_REPO: canary-demo
_VERSION: "2.0.1"
_TAG: canary
_FAIL_RATE: "0"
serviceAccount: projects/${PROJECT_ID}/serviceAccounts/canary-builder@${PROJECT_ID}.iam.gserviceaccount.com
options:
logging: CLOUD_LOGGING_ONLY
timeout: 1800s
steps:
- id: build
name: gcr.io/cloud-builders/docker
args: ["build", "-t", "${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_SERVICE}:${_VERSION}", "app"]
- id: push
name: gcr.io/cloud-builders/docker
args: ["push", "${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_SERVICE}:${_VERSION}"]
- id: snapshot-stable
name: gcr.io/google.com/cloudsdktool/cloud-sdk
entrypoint: bash
args: ["ci/snapshot.sh", "${_SERVICE}", "${_REGION}"]
- id: shadow-deploy
name: gcr.io/google.com/cloudsdktool/cloud-sdk
entrypoint: gcloud
args:
- run
- deploy
- ${_SERVICE}
- --region=${_REGION}
- --image=${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_SERVICE}:${_VERSION}
- --no-traffic
- --tag=${_TAG}
- --update-env-vars=APP_VERSION=${_VERSION},FAIL_RATE=${_FAIL_RATE}
- id: canary-and-promote
name: gcr.io/google.com/cloudsdktool/cloud-sdk
entrypoint: bash
args: ["ci/canary.sh", "${_SERVICE}", "${_REGION}", "${_TAG}"]Make the scripts executable before you submit, or call them through bash as the YAML does. The builder image ships Python 3, so gate.py and the inline JSON parsing run without extra installs.
Step 12: Run a bad release, then a good one
Submit the pipeline twice. The first run ships the faulty build with a nonzero FAIL_RATE and should die at the shadow gate. The second ships the fixed build with FAIL_RATE at zero and should walk all the way to 100%. Seeing both outcomes is the real test that the guardrails work.
chmod +x ci/*.sh
# Run 1: the faulty release
gcloud builds submit . --config cloudbuild.yaml \
--substitutions=_VERSION=2.0.0,_FAIL_RATE=0.2
# Run 2: the healthy release
gcloud builds submit . --config cloudbuild.yaml \
--substitutions=_VERSION=2.0.1,_FAIL_RATE=0The first build should end with a failure, and its log should resemble the excerpt below. Production never moved, so users saw nothing.
Stable: ledger-api-00001-abc | Candidate: ledger-api-00002-def | URL: https://canary---ledger-api-xxxxxxxxxx-uc.a.run.app
== Shadow gate (0 percent traffic) ==
{"requests": 200, "errors": 38, "error_rate": 0.19, "p95_ms": 127.3}
GATE FAILED
ROLLBACK: sending 100 percent back to ledger-api-00001-abc
ERROR: build step 4 "gcr.io/google.com/cloudsdktool/cloud-sdk" failed: step exited with non-zero status: 1The second build is the satisfying one. It should print three stage banners, three passing gates, and the final promotion.
== Shadow gate (0 percent traffic) ==
{"requests": 200, "errors": 0, "error_rate": 0.0, "p95_ms": 121.6}
GATE PASSED
== Stage: 5 percent to ledger-api-00003-ghi ==
Error log entries in last 60s: 0
== Stage: 25 percent to ledger-api-00003-ghi ==
Error log entries in last 60s: 0
== Stage: 50 percent to ledger-api-00003-ghi ==
Error log entries in last 60s: 0
== Promote: 100 percent to ledger-api-00003-ghi ==
Promotion complete.Shift Traffic by Hand and Roll Back in Seconds
Pipelines fail at awkward moments, and sometimes you want to drive the release yourself. Every action in canary.sh is a single gcloud command, so you can replay any stage from your terminal. Knowing these commands also makes incident response faster, because the fastest rollback is one you have already typed before.
# See the current split and every tag
gcloud run services describe "$SERVICE" --format='yaml(status.traffic)'
# Send 10 percent to the candidate, 90 percent to the stable revision
gcloud run services update-traffic "$SERVICE" \
--to-revisions ledger-api-00001-abc=90,ledger-api-00003-ghi=10
# Same idea using the tag instead of the revision name
gcloud run services update-traffic "$SERVICE" --to-tags canary=10
# Instant rollback: everything back to the stable revision
gcloud run services update-traffic "$SERVICE" \
--to-revisions ledger-api-00001-abc=100
# Remove the tag once the release is finished
gcloud run services update-traffic "$SERVICE" --remove-tags canaryCloud Run applies a traffic change in seconds, and it does so without restarting instances. That speed is what makes a canary safe. If the dashboard turns red at 5%, a single command ends the incident, and only about one request in twenty ever saw the bad build. When you assign less than 100% in total, Cloud Run spreads the remainder across existing revisions in proportion to their current share, which is another reason to name both revisions explicitly.
Choosing stage sizes and soak times
The defaults of 5, 25, 50 and 100 work for a service with steady traffic. Low-traffic services need different numbers, because a 5% slice of ten requests per minute is half a request per minute, which proves nothing. The table below suggests starting points by traffic level. Treat them as defaults to adjust after a few releases, not as rules.
| Service traffic | Suggested stages | Soak per stage | Gate sample size | Notes |
|---|---|---|---|---|
| Under 10 requests per minute | Shadow, then 50%, then 100% | 5 to 10 minutes | 500 requests | Small slices produce no signal, so lean on the shadow gate |
| 10 to 500 requests per minute | 10%, 50%, 100% | 5 minutes | 500 to 1,000 | Fewer stages, longer soak |
| 500 to 5,000 requests per minute | 5%, 25%, 50%, 100% | 2 to 5 minutes | 1,000 to 5,000 | The tutorial defaults fit here |
| Over 5,000 requests per minute | 1%, 5%, 25%, 50%, 100% | 1 to 3 minutes | 5,000 and up | A 1% slice still yields statistically useful volume |
These ranges come from engineering judgment, not from a Google benchmark, so validate them against your own error budgets. The pipeline reads its stages from the STAGES variable, which means you can change them per environment without editing the script. Set STAGES="10 50" in a step’s env block and the loop adapts.
Watch the Canary in Cloud Logging and Monitoring
A gate script samples synthetic requests. Real users reveal different failure modes, so the pipeline also counts error logs for the new revision. You can run the same query by hand in the Logs Explorer, or from the terminal. Both approaches filter on the revision name that Cloud Run attaches to every log line.
gcloud logging read \
'resource.type="cloud_run_revision"
AND resource.labels.service_name="ledger-api"
AND resource.labels.revision_name="ledger-api-00003-ghi"
AND severity>=ERROR' \
--freshness=10m --limit=20 \
--format='table(timestamp, jsonPayload.path, jsonPayload.status)'For metrics, Cloud Monitoring exposes the run.googleapis.com/request_count metric with a response_code_class label and a revision_name label. Chart the 5xx share per revision on one graph, and the canary and stable lines sit side by side. If the canary line climbs while the stable line stays flat, you have your answer without reading a single log.
Google’s observability documentation currently uses several names for related features, which confuses newcomers. Cloud Logging stores the logs, Cloud Monitoring holds metrics and alerts, and Observability Analytics is the SQL-style query layer that was previously called Log Analytics. Some older pages still use the old name. Google also describes Cloud Monitoring as gaining support for OpenTelemetry Protocol (OTLP) metrics in late October 2026, though that item is reported in a single Google-sourced announcement, so check the release notes before you build on it.
Build an alert that mirrors the gate
The pipeline only watches during the rollout window. An alert policy keeps watching afterward. Create a policy on the request-count metric, grouped by revision, that fires when the 5xx ratio for any revision passes your error budget for five minutes. Route it to the same channel your on-call engineer already reads. If the alert fires after promotion, you still have the previous revision available for an instant rollback, because Cloud Run keeps old revisions around until you delete them.
Seven Common Pitfalls When Canarying on Cloud Run
Most canary failures come from a handful of repeat mistakes. Each of the seven below has burned a real team, and each takes under a minute to prevent once you know about it.
- Promoting with
--to-latest. The flag puts the service in follow-the-latest mode. Every later deploy then takes 100% of traffic at once, which silently turns your canary pipeline into a big-bang pipeline. Promote by revision name instead. - Using
--set-env-varson a deploy. It replaces the entire environment of the new revision. A canary that lost its database URL fails every request and looks like a code bug. Use--update-env-vars. - Treating a tag as a percentage. A tag only creates a routable URL. A tagged revision with no traffic assignment receives zero production requests, which is exactly why the shadow stage works, but it also means tagging alone does not start a canary.
- Ignoring cold starts in the gate. A fresh revision has no warm instances, so the first requests are slow and can breach a tight latency limit. Send a short warm-up burst before you measure, or set a minimum instance count on the candidate.
- Gating on too few requests. Two hundred requests cannot detect a 0.5% regression. Size the sample to the effect you need to see, and fail closed when the sample is too small.
- Forgetting that the split is per request. Cloud Run assigns each request to a revision independently, so one user can bounce between versions during a rollout. If your release changes an API contract, a user may see inconsistent behavior. Make versions backward compatible, or add session affinity where it fits.
- Shipping a breaking database migration. The stable and canary revisions run at the same time against the same database. A migration that drops a column breaks the stable revision the moment it runs. Use expand-and-contract migrations in two separate releases.
A related trap is running two pipelines at once. Both would use the tag canary, and the second deploy would move the tag away from the first revision mid-test. Either serialize releases with a concurrency lock, or make the tag unique per build by deriving it from the build ID or commit SHA.
Troubleshooting: 12 Errors and Their Fixes
Canary pipelines touch IAM, networking, build settings and traffic routing, so the errors come from several directions. The table maps the symptoms you are most likely to hit to their causes and fixes. Scan it first when a run fails, because nine times out of ten the answer sits in one of these rows.
| # | Symptom | Likely cause | Fix |
|---|---|---|---|
| 1 | PERMISSION_DENIED mentioning iam.serviceaccounts.actAs | Builder account lacks serviceAccountUser on the runtime account | Re-run the last binding in Step 3 |
| 2 | Build fails with a logs bucket or logging error | Custom build service account without a logging option | Set options.logging: CLOUD_LOGGING_ONLY and grant logging.logWriter |
| 3 | Tagged URL returns 403 | Service is private, or allow-unauthenticated was blocked by an organization policy | Send an identity token (see Advanced Tips) and grant run.invoker |
| 4 | Tagged URL shows a generic not-found page | Wrong host format, or the tag was moved or removed | Copy the URL from describe output and remember the triple hyphen |
| 5 | Revision fails with a container failed to start and listen on the port error | App ignores the PORT variable or binds to localhost only | Read process.env.PORT and listen on all interfaces |
| 6 | Shadow gate passes, but the 5% stage fails | Cold starts, or production data exposes a bug the synthetic path misses | Add a warm-up burst, raise the gate sample, test real endpoints |
| 7 | gcloud logging read returns permission denied | Builder account cannot read logs | Grant roles/logging.viewer |
| 8 | update-traffic reports a revision not found | Typo in the name, or the revision was deleted | Run gcloud run revisions list and copy the exact name |
| 9 | A --no-traffic deploy still takes all traffic | Service is in follow-the-latest mode from an earlier --to-latest | Pin traffic with --to-revisions NAME=100 |
| 10 | Build stops after 10 minutes | Default build timeout is shorter than your soak windows | Set timeout: 1800s in the YAML |
| 11 | Snapshot step says no revision holds 100 percent | A previous failed run left a split such as 90 and 10 | Reset with update-traffic --to-revisions STABLE=100 |
| 12 | Push fails with denied on Artifact Registry | Missing writer role, or a region mismatch in the image path | Grant artifactregistry.writer and match the repo region |
When none of these fit, read the failing step’s log in the Cloud Build console first, then run the exact failing command from your own terminal. If it works there, the problem is the builder account’s permissions. If it fails the same way, the problem is in the command or the service configuration. That two-minute test splits most outages cleanly in half.
Advanced Tips for Production Canary Pipelines
Gate private services with identity tokens
Production services should not be public. Remove --allow-unauthenticated, grant the builder account roles/run.invoker, and pass a token to the gate. The script already reads a TOKEN variable, so the only change is minting one with the service URL as the audience. Test this locally before you rely on it in Cloud Build, because the way you mint tokens differs between your user account and a service account.
# Local test with your own identity
TOKEN=$(gcloud auth print-identity-token) \
python3 gate/gate.py --url "$CANARY_URL"
# In CI, mint the token for the builder account
TOKEN=$(gcloud auth print-identity-token \
--impersonate-service-account="$BUILDER_SA" \
--audiences="$SERVICE_URL") \
python3 gate/gate.py --url "$CANARY_URL"Compare the canary against the baseline, not a fixed number
A fixed 1% error ceiling punishes the canary for problems the stable revision shares, such as a flaky downstream API on a bad afternoon. A relative gate removes that noise. Run the same request mix against both the stable tag and the canary tag, then fail only when the canary’s error rate exceeds the baseline’s by a set margin. To do this, tag the stable revision as well (for example --tag stable), then call gate.py on each URL and compare the two JSON summaries.
Keep infrastructure as code from fighting your pipeline
If Terraform or OpenTofu owns the Cloud Run service, a later apply can reset the traffic block and undo a canary in progress. Tell the tool to ignore the fields your pipeline manages, namely traffic and the image. Our Terraform on Google Cloud tutorial sets up the surrounding project and state, which makes a good base for this change.
Add human approval where the stakes are high
Google’s documentation does not show a built-in approval step for Cloud Run traffic promotion, so the pipeline above promotes automatically. For regulated systems, you can insert a manual gate in two ways. One option is to stop at 50% and require a second, manually triggered build for the final promotion. The other is Cloud Deploy, which lists Cloud Run as a deployment target and adds approvals between environments, at the cost of one more service to learn.
Other platforms and multi-region setups
The same ideas carry over to other clouds. Azure Container Apps uses a revision model with traffic weights, and our Azure Container Apps setup guide shows where those controls live. For a broader view of how the providers differ, see the AWS versus Google Cloud comparison. On the Google side, Google announced multi-region readiness probes and automated regional failover improvements in July 2026, according to coverage of its product updates. If you run in several regions, repeat the canary in one region first and treat it as a regional canary before the global rollout.
One boundary to respect: Cloud Run now spans services, jobs, worker pools and the newer Cloud Run instances. Instances are described as singleton, long-running runtimes rather than autoscaling services, so revision traffic splitting is a feature of services and not something to expect from them. For more guides in this area, browse our cloud computing coverage.
Complete Project Checklist, Cost and Cleanup
Before you call the project done, check that every file from the layout exists and runs. The one-shot sequence below recreates the whole demo from a clean shell, assuming Steps 1 to 3 are complete. It builds the baseline, ships a bad release that must fail, and then ships a good one that must reach 100%.
#!/usr/bin/env bash
# run-demo.sh : baseline, bad release, good release
set -euo pipefail
IMAGE_BASE="${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO}/${SERVICE}"
gcloud builds submit app --tag "${IMAGE_BASE}:1.0.0"
gcloud run deploy "$SERVICE" --image "${IMAGE_BASE}:1.0.0" \
--service-account "$RUNTIME_SA" --allow-unauthenticated \
--update-env-vars APP_VERSION=1.0.0
chmod +x ci/*.sh
# Bad release: this build is expected to FAIL and roll back
gcloud builds submit . --config cloudbuild.yaml \
--substitutions=_VERSION=2.0.0,_FAIL_RATE=0.2 || echo "Bad release stopped as designed"
# Good release: this build is expected to reach 100 percent
gcloud builds submit . --config cloudbuild.yaml \
--substitutions=_VERSION=2.0.1,_FAIL_RATE=0
curl -s "$(gcloud run services describe "$SERVICE" --format='value(status.url)')/version"The final call should print the new version and the name of the newest revision. If it prints the old version, the promotion never ran, so read the last build log.
What the demo costs
Cloud Run services are pay-per-use and scale to zero when idle, so the demo’s compute bill is small. Two other lines deserve attention. Cloud Build bills for the minutes a build runs, and your soak windows count because the build sits waiting. Artifact Registry charges for stored images. Google has not published a full, current price table in the sources we checked, so use the Google Cloud pricing calculator for your region rather than trusting a number from a blog post, including this one.
Google also said monthly Cloud Run spending caps are coming, with resources deactivated when a cap is reached. As of October 28, 2026, the announcements describe the feature as forthcoming, so do not treat it as a safety net yet. Set Cloud Billing budget alerts today, and compare the reporting tools in our cloud cost tool comparison if you manage more than one provider.
Step 13: Clean up revisions, tags and images
The last step keeps the bill flat. Remove the tag first, delete revisions you no longer need to roll back to, and prune old images. For a throwaway demo, tear everything down in one pass.
# Routine hygiene after a successful release
gcloud run services update-traffic "$SERVICE" --remove-tags canary
gcloud run revisions list --service "$SERVICE"
gcloud run revisions delete ledger-api-00002-def --quiet
gcloud artifacts docker images delete "${IMAGE_BASE}:2.0.0" --delete-tags --quiet
# Full teardown of the demo
gcloud run services delete "$SERVICE" --quiet
gcloud artifacts repositories delete "$REPO" --location="$REGION" --quiet
gcloud iam service-accounts delete "$RUNTIME_SA" --quiet
gcloud iam service-accounts delete "$BUILDER_SA" --quietCloud Run refuses to delete a revision that still receives traffic, which protects you from deleting the stable one by accident. For ongoing cleanup, Artifact Registry supports cleanup policies that expire untagged or old images automatically, so you do not have to script pruning forever.
FAQ: Canary Deployment on Google Cloud Run
What is a canary deployment on Cloud Run?
It is a release method where a new revision first receives a small percentage of production traffic while the previous revision serves the rest. You raise the share in stages if metrics stay healthy, and send it back to zero if they do not.
Is a shadow revision the same as traffic mirroring?
No. The shadow pattern deploys a revision with no production traffic and tests it through its tagged URL. Based on the official material we reviewed, Cloud Run does not offer a first-party mode that copies live requests to a second revision.
Do I need Cloud Build, or can I use another CI system?
Any CI system works. Cloud Build only orchestrates. The deploy, tag and traffic commands are ordinary gcloud calls, so another CI system can run the same scripts with a suitable service account.
Does Cloud Run split traffic per user or per request?
Per request. A single user can land on different revisions during a rollout, so keep versions compatible with each other and consider session affinity if your release changes behavior users can see.
How fast can I roll back?
Changing the traffic split takes effect within seconds, because the old revision’s container image and configuration already exist. One update-traffic command that sets the stable revision to 100% completes the rollback.
What is the difference between a revision tag and a label?
A tag gives a revision its own URL and lets you reference it in traffic commands. A label is metadata for organizing resources, for example in billing reports. They serve different purposes and are unrelated in practice.
Does this work for Cloud Run functions?
Cloud Run functions is the current name for what was Cloud Functions, and it runs on the Cloud Run platform. Check Google’s current documentation for which traffic-splitting options your function type supports before you copy this pipeline over.
