Coder Legion - Recent posts tagged cicd

Segurança em CI/CD: a importância de estudar, auditar e aprender com pipelines de bons projetos

Fri, 30 Oct 2026 12:10:32 +0000

Pipelines de CI/CD são frequentemente tratados como simples mecanismos de automação: executam testes, compilam aplicações, geram artefatos e realizam deploys. Entretanto, eles também representam uma das superfícies de ataque mais importantes de um ambiente de desenvolvimento moderno.

Um pipeline possui acesso a repositórios, tokens, credenciais, registries, ambientes de produção, dependências externas e, em muitos casos, permissões para modificar recursos de infraestrutura. Isso significa que uma falha na configuração de um workflow pode ultrapassar os limites do processo de build e comprometer toda a cadeia de entrega de software.

Por esse motivo, estudar a segurança de pipelines deve fazer parte da rotina de engenharia, assim como revisar código, analisar dependências e avaliar configurações de infraestrutura.

Mais importante do que simplesmente executar ferramentas de análise é compreender por que determinada configuração representa um risco, em quais condições ela pode ser explorada e quais mecanismos realmente mitigam esse risco.

1. Por que estudar pipelines é tão importante?

Um pipeline inseguro pode transformar uma alteração aparentemente simples em um vetor de comprometimento.

Considere alguns exemplos:

Um Pull Request pode introduzir código executado em um runner que possui acesso a credenciais.
Uma interpolação insegura de contextos do GitHub pode permitir injeção de comandos no shell.
Uma dependência comprometida pode executar código durante a instalação.
Um token com permissões excessivas pode permitir alterações não autorizadas no repositório.
Uma publicação sem mecanismos adequados de autorização pode disponibilizar uma versão não aprovada de um pacote.
Um workflow mal configurado pode executar código de terceiros com privilégios superiores aos necessários.

Esses cenários demonstram que o pipeline não é apenas uma ferramenta operacional. Ele faz parte da infraestrutura de segurança da organização.

A segurança precisa considerar todo o ciclo de execução: desde o evento que dispara o workflow até as permissões disponíveis, dependências instaladas, artefatos produzidos e operações realizadas.

2. Utilize scanners, mas não terceirize o julgamento técnico

Existem ferramentas especializadas em identificar configurações inseguras em workflows.

Algumas opções relevantes que eu utilizo no dia a dia para scans do pipeline:

zizmor: analisa workflows do GitHub Actions e identifica problemas como template injection, permissões inadequadas, uso inseguro de eventos e instalações de dependências fora de manifests.
Checkov: realiza análise estática de configurações de infraestrutura como código e também identifica problemas em configurações de CI/CD.
actionlint: valida a sintaxe, expressões, referências e possíveis problemas estruturais em workflows do GitHub Actions.
OpenSSF Scorecard: avalia práticas de segurança de projetos e repositórios, incluindo dependências, permissões, manutenção e configuração de workflows.
Trivy: além de analisar imagens e dependências, consegue avaliar configurações e arquivos de infraestrutura.

Essas ferramentas são importantes porque automatizam a identificação de padrões conhecidos de risco.

Entretanto, existe uma diferença fundamental entre encontrar uma configuração potencialmente insegura e demonstrar que ela é explorável.

Um scanner identifica padrões. O engenheiro precisa compreender o contexto.

Exemplo: template injection

Considere:

- name: Check changelog
  run: scripts/check-changelog-frozen.sh "origin/${{ github.base_ref }}"

O scanner pode identificar uma possível injeção de comandos porque o contexto é interpolado diretamente no código executado pelo shell.

O risco existe porque o GitHub Actions realiza a expansão da expressão antes de executar o script.

Entretanto, é necessário entender a origem do dado.

github.base_ref representa a branch de destino do Pull Request, enquanto github.head_ref representa a branch de origem.

Um colaborador externo normalmente controla o nome da própria branch, mas não controla livremente a branch de destino.

Portanto, embora exista um padrão vulnerável, a explorabilidade depende das permissões e das condições do workflow.

A correção recomendada é transportar o valor por uma variável de ambiente:

- name: Check changelog
  env:
    BASE_REF: ${{ github.base_ref }}
  run: scripts/check-changelog-frozen.sh "origin/$BASE_REF"

O valor passa a ser tratado como dado, em vez de ser inserido diretamente no código interpretado pelo shell.

Esse exemplo demonstra por que é necessário estudar os findings: o scanner identifica corretamente o padrão, mas a análise de impacto exige conhecimento sobre os contextos do GitHub Actions.

3. Aprenda a diferenciar vulnerabilidades reais de findings parcialmente mitigados

Nem todo alerta representa uma vulnerabilidade diretamente explorável.

Um exemplo interessante é a instalação de dependências durante o pipeline.

Considere:

env:
  CLAUDE_CODE_VERSION: "2.1.296"
  CODEX_VERSION: "0.162.1"

E posteriormente:

- name: Install CLIs
  run: |
    npm install --global \
      "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" \
      "@openai/codex@${CODEX_VERSION}"

Um scanner como o zizmor pode identificar o padrão adhoc-packages, pois os pacotes são instalados globalmente sem um manifesto e lockfile controlando a instalação.

Entretanto, existe uma particularidade: as versões principais já estão explicitamente fixadas.

Isso significa que o workflow não está simplesmente instalando a última versão disponível a cada execução.

Ainda existe uma fragilidade: a árvore completa de dependências não está necessariamente reproduzível, pois as dependências transitivas não são controladas por um lockfile versionado.

É importante distinguir:

Versão principal fixada: controla qual versão direta será solicitada.
Manifesto: declara as dependências do projeto.
Lockfile: registra a resolução das dependências e suas integridades.
Instalação reproduzível: utiliza a resolução registrada para reconstruir o ambiente.

A ausência de lockfile continua sendo um finding válido, mas não significa automaticamente que existe uma instalação de versão flutuante ou que um pacote esteja comprometido.

O objetivo da análise não é eliminar alertas indiscriminadamente, mas determinar quais riscos permanecem e quais controles já existem.

4. Estude como projetos maduros resolvem problemas de segurança

Uma das formas mais eficientes de evoluir na segurança de pipelines é estudar workflows de projetos open source bem mantidos.

Em vez de começar sempre do zero, investigue como projetos com grande exposição pública estruturam seus processos de CI/CD.

Alguns exemplos interessantes para estudo:

Kubernetes: organização de workflows, testes, automação e governança de contribuições.
Terraform: validação, testes, releases e publicação de artefatos.
Trivy: análise de segurança, integração de scanners e automação de releases.
OpenTelemetry: testes distribuídos, validação de componentes e publicação de versões.
actions/checkout: implementação e manutenção de GitHub Actions.
Sigstore: assinatura, verificação e estabelecimento de confiança em artefatos.

Ao estudar esses projetos, não observe apenas quais actions são utilizadas. Investigue as decisões arquiteturais.

Pergunte:

Como o workflow é disparado?
Ele aceita Pull Requests de forks?
Quais permissões são concedidas ao GITHUB_TOKEN?
Existem secrets disponíveis durante a execução?
Como as dependências são instaladas?
As GitHub Actions são referenciadas por tags ou hashes SHA completos?
Como os artefatos são assinados e verificados?
Como as publicações são autorizadas?
Existem ambientes protegidos e aprovações humanas?
Como o projeto separa testes, validação e publicação?

O objetivo é compreender o motivo de cada decisão, e não simplesmente copiar configurações.

Uma configuração segura em determinado contexto pode ser inadequada em outro.

5. Construa uma rotina de auditoria de pipelines

A análise não deve acontecer somente quando um scanner encontra um problema.

Uma rotina de auditoria pode ser incorporada ao próprio processo de engenharia.

Uma abordagem prática:

Etapa 1 — Inventário

Mapear todos os workflows existentes:

CI.
CD.
Release.
Publicação de pacotes.
Provisionamento de infraestrutura.
Automação de dependências.
Segurança e compliance.

Etapa 2 — Análise automatizada

Executar ferramentas como:

zizmor .github/workflows

actionlint

checkov -d .github/workflows

scorecard --repo=github.com/ORG/REPOSITORY

Os comandos devem ser ajustados à versão e à forma de instalação de cada ferramenta.

Etapa 3 — Investigação dos findings

Para cada alerta:

Identificar o trecho responsável.
Entender a regra acionada.
Identificar a origem dos dados.
Verificar se existe controle compensatório.
Avaliar a possibilidade de exploração.
Determinar o impacto potencial.
Definir a correção ou justificar uma exceção.

Etapa 4 — Correção

Priorizar problemas que permitam execução de código não confiável, exposição de credenciais, elevação de privilégios e comprometimento de releases.

Etapa 5 — Prevenção

Incorporar os scanners ao próprio CI para impedir que configurações inseguras sejam introduzidas novamente.

6. Crie um laboratório utilizando projetos reais

Uma estratégia especialmente interessante é escolher projetos open source e realizar auditorias independentes em seus workflows.

Não é necessário encontrar uma vulnerabilidade para que o estudo seja útil.

O exercício consiste em:

Selecionar um repositório ativo e bem mantido.
Baixar ou clonar seus workflows.
Executar os scanners.
Investigar os findings.
Consultar a documentação das regras acionadas.
Comparar a implementação com as recomendações oficiais.
Identificar controles compensatórios.
Propor correções.
Registrar os resultados em um pequeno relatório técnico.

É possível inclusive comparar diferentes projetos para entender como cada equipe resolve o mesmo problema.

Por exemplo: como diferentes projetos implementam Trusted Publishing, como controlam dependências de terceiros ou como protegem workflows executados por Pull Requests externos.

Esse exercício desenvolve uma habilidade importante: sair da posição de alguém que apenas corrige alertas e passar a compreender a arquitetura de segurança do processo de entrega.

Segurança de CI/CD exige conhecimento sobre GitHub Actions, shell, permissões, autenticação, dependências, infraestrutura e supply chain security.

É justamente por isso que estudar workflows de projetos maduros é tão valioso: eles oferecem exemplos concretos de decisões de engenharia, problemas reais e estratégias de mitigação que podem ser adaptadas a outros ambientes.

Conclusão

Scanners são instrumentos de diagnóstico, não substitutos do conhecimento técnico.

Executar um scanner e aplicar todas as correções sugeridas sem entender os findings pode gerar configurações desnecessárias, exceções mal justificadas e até novos problemas.

O caminho mais consistente é combinar análise automatizada, estudo da documentação, investigação técnica e comparação com implementações de projetos reconhecidos.

Estudar pipelines de bons projetos é uma forma de aprender com decisões de engenharia que já foram colocadas à prova.

Afinal, um pipeline seguro não é aquele que simplesmente passa em todas as verificações. É aquele cuja arquitetura foi compreendida, cujos riscos são conhecidos e cujos mecanismos de proteção foram implementados de maneira consciente.

Flyway Migration: From Legacy Database to CI/CD

Thu, 29 Oct 2026 12:05:30 +0000

Most teams think of Flyway only when they need to manage ongoing database schema changes. But what if your project already has a complete, stable schema and you’re only now introducing version control? Or what if you want to incorporate your schema into a CI/CD pipeline for reliable, reproducible deployments?

In this guide, I’ll walk you through a real-world scenario where we used Flyway to baseline an existing MySQL database and then extended it to support version-controlled schema migrations in CI/CD, without disrupting production.

What is Flyway?

Flyway is a simple but powerful tool for managing database migrations. It applies versioned SQL scripts to your database and tracks them in a special table called flyway_schema_history.

Typical script names:

V1__initial_schema.sql
V2__add_orders_table.sql
V3__rename_column.sql

Flyway ensures these scripts run only once and in the correct order. It also integrates well with CI/CD tools, making it ideal for automated environments.

Use Case: Existing Schema with No Migrations (Yet)

In our project, we had:

A production MySQL database with dozens of tables
No migration history
No current need for structural changes
A desire to version control the database and enable future migrations via CI/CD

Step 1: Baselining an Existing Schema

Dump the schema

mysqldump -u root -p --no-data your_database > V1__baseline_schema.sql

Clean the output: remove auto-increments, user references, and environment-specific SQL.

Create directory structure

mkdir -p db/sql
mv V1__baseline_schema.sql db/sql/

Configure flyway.conf

flyway.url=jdbc:mysql://your-db-host:3306/your_database
flyway.user=db_user
flyway.password=secret_password
flyway.locations=filesystem:db/sql
flyway.baselineOnMigrate=true

Run baseline

flyway baseline

This creates the flyway_schema_history table and marks version 1 as the current state without running any script.

Step 2: Add a Real Migration

After baselining, we added a real change in a new script

-- db/sql/V2__add_status_to_orders.sql
ALTER TABLE orders ADD COLUMN status VARCHAR(20) DEFAULT 'pending';

Then we ran:

flyway migrate

Result: Flyway applied version 2, recorded it in history, and our database was now safely versioned.

⚙️ Step 3: Integrate Flyway into CI/CD

You can integrate Flyway with GitHub Actions, GitLab CI, Bitbucket Pipelines, or Jenkins.

✅ Example GitHub Actions Workflow

name: Flyway Migration
on:
  push:
    branches: [ main ]

jobs:
  migrate:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
      - name: Set up Java (Flyway needs it)
        uses: actions/setup-java@v3
        with:
          distribution: 'temurin'
          java-version: '17'
      - name: Install Flyway
        run: |
          curl -L https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/10.8.1/flyway-commandline-10.8.1-linux-x64.tar.gz | tar xvz
          sudo ln -s $PWD/flyway-10.8.1/flyway /usr/local/bin/flyway
      - name: Run Flyway Migrations
        env:
          FLYWAY_URL: jdbc:mysql://your-host:3306/your_database
          FLYWAY_USER: ${{ secrets.DB_USER }}
          FLYWAY_PASSWORD: ${{ secrets.DB_PASSWORD }}
        run: |
          flyway -url=$FLYWAY_URL \
                 -user=$FLYWAY_USER \
                 -password=$FLYWAY_PASSWORD \
                 -locations=filesystem:./db/sql \
                 migrate

Benefits of This Approach

✅ You gain auditability for all database changes

✅ Developers can build local environments using the same scripts

✅ CI/CD ensures consistency across all environments

✅ Flyway’s validate command catches missing or changed scripts

✅ Schema management becomes part of your software delivery lifecycle

Potential Challenges

Problem → Fix

Flyway re-applies the baseline script
→ Use baselineOnMigrate=true and never modify a baseline
Passwords in config files
→ Use secrets and environment variables in CI/CD
Manual schema drift
→ Disable direct DB access in prod or run flyway validate periodically
Need rollback support
→ Write manual undo scripts or use point-in-time backups

Alternatives (and Why Flyway Wins)

Liquibase
✅ Pros: Advanced features, rollback support
❌ Cons: More complex, steeper learning curve

Django ORM / Alembic
✅ Pros: Built-in for app devs
❌ Cons: Not cross-language or DB-agnostic

Manual SQL or DB dumps
✅ Pros: Simple
❌ Cons: No versioning, no tracking, no automation

Flyway is a great middle-ground: powerful enough for teams, simple enough for solo devs, and perfect for CI/CD.

Final Thoughts

Flyway isn’t just for teams writing migrations every day. It’s also ideal for bringing an existing database under version control, giving your project a foundation for safe schema evolution.

We started with a baseline, added our first real migration, and then built it into our CI/CD pipeline — all in a day’s work.

If you’re managing an existing DB without schema tracking, now’s the time to fix that. Start with a baseline. Your future self will thank you.

Let’s Connect!

Have you tried Flyway for baseline-only use cases or integrated it into your CI/CD pipeline? I’d love to hear your experience in the comments.

If you found this helpful, follow me here on DEV for more DevOps and database engineering content.

⚙️From 4 Hours to 3 Minutes: Automating GCP Ingress Provisioning with Ansible

Tue, 27 Oct 2026 16:36:09 +0000

Context

Every time a new client was onboarded, their web application had to be deployed and exposed to the internet.

As part of the deployment, the infrastructure required several configuration steps across different services before the application was ready to receive traffic.

Problem

Although the deployment workflow was always the same, every new client required repeating the exact same operational tasks:

Creating GCP firewall rules
Retrieving the VM's public IP
Creating or updating Cloudflare DNS records
Installing and configuring Nginx
Generating Let's Encrypt SSL certificates
Validating the routing and HTTPS configuration

Since these tasks were performed manually for every client, the provisioning process typically took around four hours to complete. Besides being time-consuming, it also increased the likelihood of configuration mistakes and inconsistencies.

Solution

After repeating the same workflow several times, I realized the process was entirely deterministic. The only inputs that changed between deployments were values such as the domain name, backend ports, and service account.

Instead of spending hours repeating the same sequence of tasks, I automated the entire workflow with Ansible and integrated it into our CI/CD pipeline.

How the workflow looks

GitLab CI/CD
      │
      ▼
Run Ansible Playbook
      │
      ▼
Create GCP Firewall Rule
      │
      ▼
Retrieve VM Public IP
      │
      ▼
Create / Update Cloudflare DNS Record
      │
      ▼
Install Nginx
      │
      ▼
Generate HTTP Configuration
      │
      ▼
Request Let's Encrypt Certificate
      │
      ▼
Generate HTTPS Configuration
      │
      ▼
Reload Nginx
      │
      ▼
Application Available via HTTPS

A single CI/CD job orchestrates the entire workflow. Ansible handles the infrastructure configuration, DNS updates, reverse proxy setup, and SSL provisioning, making the deployment consistent and repeatable.

Implementation Highlights

Google Cloud Firewall Automation – Creates HTTP/HTTPS firewall rules through the GCP API.
Automatic Public IP Discovery – Retrieves the VM's external IP from the GCP metadata service instead of requiring manual input.
Cloudflare DNS Management – Updates DNS records automatically as part of the deployment.
Dynamic Nginx Configuration – Uses Jinja2 templates to generate HTTP and HTTPS configurations.
Automated SSL Provisioning – Obtains and configures Let's Encrypt certificates without manual intervention.
CI/CD Integration – The entire process runs automatically through GitLab CI.

Results

Before	After
~4 hours to provision and configure	~3 minutes through CI/CD
Multiple manual steps	Fully automated workflow
Manual DNS, firewall and SSL configuration	Managed by Ansible
Repetitive onboarding process	Consistent and repeatable deployment

Beyond the time savings, every deployment now follows the exact same process.

Instead of spending hours repeating the same steps, I simply trigger the pipeline and let the automation do the work.

Check the code

https://github.com/ja-lourenco/ansible-gcp-ingress-proxy

Dynamic GitHub Actions Dropdowns — A Clever Workaround Made Reusable!

Fri, 16 Oct 2026 19:57:00 +0000

GitHub Actions doesn't natively support dynamically populated dropdown options for workflow_dispatch inputs. This can become a maintenance headache, especially when managing multiple packages in an npm monorepo.

I came across a clever workaround shared by @jeevan-vj, demonstrating how to dynamically maintain dropdown options by automatically updating GitHub Actions workflow YAML files.

Inspired by that approach, I built a reusable GitHub Actions workflow that uses a dedicated WORKFLOW_TOKEN to update workflow files and automatically discover publishable packages in an npm monorepo.

Here's what my implementation adds:

✅ Automatic package discovery — Detects publishable npm workspace packages without maintaining a hardcoded list.

✅ Multi-workflow synchronization — Keeps package dropdown options consistent across multiple GitHub Actions workflows.

✅ YAML formatting preservation — Uses Python and ruamel.yaml to update workflow configurations while preserving their structure and formatting.

✅ Automatic and manual execution — Supports synchronization on push as well as manual triggering when needed.

✅ Configurable token authentication — Accepts a workflow-token secret, allowing each consuming repository to choose its own secret name.

✅ No unnecessary commits — Updates workflow files only when package options actually change.

The end result?

My release, publish, restore, reset, and release-draft workflows now automatically stay synchronized with the publishable packages in my npm monorepo.

No more manually updating package dropdowns every time a package is added, renamed, or removed.

Even better, the solution is reusable across repositories, so I have adopted the same automation in other repos without duplicating the synchronization logic.

Original inspiration — @jeevan-vj:
https://github.com/jeevan-vj/iamjeevan/blob/master/data/blog/dynamically-populating-parameter-options-in-manual-github-action.md

My reusable workflow implementation:
https://github.com/moyarich/reusable-workflows/blob/main/.github/workflows/reusable_sync-release-package-choices.yml

Real-world implementation — Workspace Tools:
https://github.com/moyarich/workspace-tools/blob/main/.github/workflows/sync-release-package-choices.yml

Huge credit to @jeevan-vj for sharing the original workaround. It's a great example of how an open-source idea can evolve into a reusable solution that benefits other developers.

How are you managing dynamic inputs in your GitHub Actions workflows? Have you found another approach?

Evaluating and Testing AI Agents in Production: How to Measure the Unpredictable

Tue, 06 Oct 2026 06:11:00 +0000

In conventional software development, the boundary between success and failure is binary, deterministic, and unambiguous. You write a discrete function, formulate a unit test with assert calculate_discount(100, 0.2) == 80, and when your continuous integration pipeline turns green, you ship to production with absolute confidence.

Yet when building an Autonomous AI Agent, that entire bedrock of certainty dissolves:

User inputs are unstructured, open-ended natural language prompts.
The underlying foundation model is fundamentally stochastic (probabilistic) by mathematical design.
The agent dynamically orchestrates multi-step Tool Calling, executes SQL queries, traverses external APIs, and autonomously decides how many intermediate reasoning cycles it requires to fulfill its objective.
Most perilously: a minor adjustment to a single phrase in your system prompt designed to improve conversational tone can silently cause the agent to omit a mandatory currency parameter in a Stripe API call or trigger an infinite retry loop in an unrelated workflow.

In 2026, following our operational deep dives into MLOps for Engineers, execution vulnerabilities in Prompt Injection, and autonomous orchestration across our Autopilot series, the software engineering industry has encountered an unavoidable truth: over 80% of enterprise AI agent initiatives stall when attempting to cross the chasm separating a flashy prototype in a Jupyter notebook from an enterprise-grade production environment.

The root cause is not an intellectual deficit in frontier reasoning engines like Gemini 3.8, Claude Fable 5.1, or GPT Sol 5.6. The root cause is the absence of a rigorous, repeatable discipline of continuous evaluation and automated testing.

How do we systematically test that which is non-deterministic by architecture?

The Collapse of Legacy Metrics and the Peril of 'Vibe Checking'

During the formative years of generative AI adoption, engineering teams leaned on methodological shortcuts that are today recognized as technically inadequate:

1. Classical NLP Metrics (BLEU, ROUGE, METEOR)

Formulated decades ago for machine translation and extractive summarization, these metrics measure raw n-gram surface overlap between generated text and a static reference string. In agentic workflows, they are blind:

An agent can deliver an answer using entirely disparate vocabulary while being technically, mathematically, and logically flawless.
Conversely, an agent can achieve a 95% word match against a reference answer while introducing a catastrophic numerical error (“Flight departs at 14:00” vs. “Flight departs at 04:00”).

2. Embedding Cosine Similarity

Evaluating accuracy by measuring semantic distance in vector space (as examined in GraphRAG) represents a dangerous engineering trap. The propositions “The agreement has been formally signed and approved” and “The agreement has not been signed or approved” share a cosine proximity exceeding 0.93, despite possessing diametrically opposed legal, operational, and financial realities.

3. 'Vibe Checking'

The most pervasive anti-pattern in modern software engineering: a developer manually runs three prompts in a terminal, skims the conversational responses, nods approvingly because the prose “sounds convincing,” and approves the Pull Request. This practice is the direct equivalent of shipping a core banking engine without unit tests simply because the ATM screen illuminated upon being plugged into the wall.

The 4-Layer Evaluation Framework for AI Agents

To guarantee operational reliability in production, an agent's execution must be deconstructed across four independent yet interlocking analytical layers:

┌─────────────────────────────────────────────────────────────┐
│  LAYER 1: Trajectory and Tool Calling Evaluation            │
│  (Tool choice accuracy, schema validation, loop detection)  │
├─────────────────────────────────────────────────────────────┤
│  LAYER 2: Environment State Testing                         │
│  (Database mutations, API idempotency, rollback checks)     │
├─────────────────────────────────────────────────────────────┤
│  LAYER 3: Calibrated Semantic Evaluation (LLM-as-a-Judge)   │
│  (CoT rubrics, RAGAS faithfulness, bias mitigation)         │
├─────────────────────────────────────────────────────────────┤
│  LAYER 4: CI/CD Pipeline Gateway and Automated Red-Teaming  │
│  (Golden datasets, pass/fail deployment thresholds, attack) │
└─────────────────────────────────────────────────────────────┘

Layer 1: Trajectory and Tool Calling Evaluation

An autonomous agent is not defined solely by its terminal text output; it is defined by the discrete intermediate steps it takes to navigate a task graph:

Tool Selection Accuracy: Did the agent select the correct tool at each decision boundary? (e.g., executing query_database rather than confabulating financial metrics from internal parametric weights).
Schema and Parameter Validation: Did the agent structure the JSON payload with strict type conformity? Did it honor enumerated constraints and required arguments?
Step-to-Goal Ratio & Loop Detection: If an agent suddenly requires twelve conversational rounds to complete a task that typically demands three, the system has suffered a silent regression that quadruples latency and inference token expenditure. The evaluation harness must programmatically flag unproductive retry cycles before exhaustively consuming budget caps.

Layer 2: Environment State Testing (State Assertions)

The only immutable truth in computer software is the physical state mutation an application imparts to its external environment. In this layer, we recover deterministic certainty:

If an agent outputs: “I have successfully cancelled your recurring subscription and issued a full refund,” the test suite does not evaluate the agent's prose. The test queries the staging database or the mock Stripe gateway directly:
```
assert user.subscription.status == "cancelled"
assert refund_transaction.amount == 49.99
```
Ephemeral Sandbox Isolation: Autonomous agents must be evaluated against disposable containerized sandboxes (ephemeral Docker clusters) where outbound HTTP mutations are intercepted, logged, and validated for strict idempotency and zero unintended side-effects.

Layer 3: Calibrated Semantic Evaluation (LLM-as-a-Judge)

When evaluating qualitative attributes (conversational empathy, synthesis completeness, or explanatory precision), we deploy another foundation model as an evaluator. However, an uncalibrated LLM judge introduces systemic risk:

Research demonstrates that LLM evaluators suffer from three documented cognitive biases:

Position Bias: In pairwise A/B benchmarks, LLM evaluators consistently favor whichever candidate response is presented first.
- Mitigation: Implement automated swap-testing (running dual passes with inverted candidate ordering) and averaging score distributions.
Verbosity Bias: Foundation models routinely award higher quality scores to longer, ornate prose, even when it harbors factual inaccuracies.
- Mitigation: Normalize text length constraints within the rubric and programmatically penalize unprompted redundancy.
Sycophancy: Evaluator models tend toward lenient, uncritical scoring to avoid conflict.
- Mitigation: Enforce Chain-of-Thought (CoT) Rubrics requiring the judge to cite explicit textual evidence before emitting a discrete numerical rating (1 to 5).

For knowledge-intensive architectures, we integrate the standardized RAGAS metric suite:

Faithfulness: Is every factual claim in the response directly grounded in retrieved context? (Mathematical hallucination detection).
Answer Relevance: Does the generated output directly address the user's core intent without thematic drift?

Layer 4: CI/CD Pipeline Gateway and Automated Red-Teaming

Evaluation loses all utility if conducted as a quarterly manual audit. It must be codified into the continuous delivery pipeline:

Curated Golden Datasets: Version-controlled, immutable benchmark suites containing hundreds of production edge cases verified by human domain experts.
Synthetic Adversarial Generation: Deploying specialized attacker agents (Adversarial Agents) that intentionally inject typos, ambiguity, contradictory instructions, and Prompt Injection payloads to test system resilience prior to merge.
Automated Deployment Gates: Within GitHub Actions or GitLab CI, if a proposed prompt revision induces a >2% drop in Tool Calling precision or a >0.5% degradation in factual faithfulness, the Pull Request is automatically blocked.

Offline Evaluation vs. Online Telemetry

Testing autonomous agents does not terminate at deployment; it transitions into a closed-loop observability pipeline:

Dimension	Offline Evaluation (Pre-Deployment)	Online Monitoring (Production)
Environment	Controlled sandboxes with synthetic mocks and test databases.	Live customer traffic and production microservice APIs.
Data Volume	Hundreds or thousands of curated test cases (Golden Sets).	Millions of real-time multi-turn interactions.
Primary Metrics	Schema conformance, branch coverage, prompt regression rate.	P95/P99 latency, tool failure rate, cost per trajectory, explicit feedback (thumbs up/down).
Core Tooling	Pytest, RAGAS, DeepEval, Promptfoo, custom eval harnesses.	OpenTelemetry, LangSmith, Arize Phoenix, Datadog LLM Observability.
Primary Objective	Prevent functional and behavioral regressions before merging to main.	Detect data drift, anomalous execution loops, and security abuse in real time.

Regulatory Imperative: Compliance Under the EU AI Act

This engineering rigor has evolved from an architectural best practice into an uncompromising legal obligation.

Under the EU AI Act, high-risk enterprise AI systems must satisfy formal technical governance standards:

Article 15 (Accuracy, Robustness, and Cybersecurity): Formally requires that AI systems be continuously evaluated against performance degradation, unexpected environmental shifts, and adversarial exploitation throughout their lifecycle.
Traceability and Audit Logging (Article 12): Mandates automated, high-fidelity logging of all intermediate tool invocations, parameters, and decision branches to permit independent external forensics.

Enterprises deploying unmonitored AI agents governed solely by manual inspections face not only catastrophic operational vulnerabilities, but also severe statutory fines for lack of demonstrably verifiable technical oversight.

5 Practical Commandments for AI Agent Engineers

If you are architecting and shipping autonomous agentic systems in production, implement these five rules immediately:

1. Never Touch a Prompt Without Running a Regression Suite

Uncalibrated prompt engineering is the contemporary equivalent of patching binary code directly on a live production server. Every prompt modification must run against an automated regression harness.

2. Decouple Reasoning Evaluation from State Verification

Never ask an agent if it completed a workflow successfully: verify the execution yourself by programmatically inspecting target database tables and microservice state machines.

3. Periodically Calibrate LLM Judges with Human Reviewers

Uncalibrated evaluator models suffer from optimistic drift. Calculate the inter-rater agreement coefficient (Cohen’s Kappa) quarterly between your automated LLM judges and senior human engineers.

4. Benchmark Cost and Latency Alongside Quality

An agent boasting 98% factual precision that consumes 45 seconds and $0.35 in inference tokens per call is commercially dead on arrival. Engineering evaluation must treat Quality, Latency, and Cost as a unified trade-off triangle.

5. Embed Automated Red-Teaming into CI/CD

The worst venue to discover an agent leaks sensitive secrets or allows prompt hijacking is when an external user posts the exploit on social media. Deploy adversarial agents designed to systematically attack your models on every commit.

Conclusion

The maturity of an engineering discipline is not measured by the audacity of its prototypes, but by the repeatability and rigor of its validation methodologies.

Deploying autonomous AI agents without a multi-layer evaluation framework is like constructing a modern suspension bridge without testing wind resistance and structural load tolerances. Foundation reasoning models will continue their rapid march forward, but enterprise competitive advantage will not belong to whoever deploys the largest model; it will belong to whoever builds the most resilient, automated evaluation infrastructure to deploy intelligence with uncompromising confidence.

Sources of Interest:

New to Bitrise CodePush? Here's What I Wish I Knew Before Setting It Up

Tue, 22 Sep 2026 15:32:27 +0000

New to Bitrise CodePush? Here's What I Wish I Knew Before Setting It Up

If you've never used CodePush before, the idea sounds almost too convenient.

You find a bug in a React Native app, fix the JavaScript, and instead of waiting for another App Store or Google Play release, you can push the update directly to users.

That's the part that got my attention.

But once I actually started setting it up, I realized there are quite a few things that aren't obvious from that one-line explanation.

So if you're new to Bitrise CodePush, here's what I learned while going through the setup.

First: What Does CodePush Actually Update?

The easiest way to think about CodePush is:

It updates the JavaScript and assets in your React Native app without requiring a new store release.

So if you've fixed:

A JavaScript bug
Some text or copy
A small UI issue
Another change that doesn't require native code

You can potentially deliver that fix without submitting a new binary to the App Store or Google Play.

But there's an important boundary.

If you change native code, CodePush doesn't magically bypass the app stores.

You'll still need a normal app release.

That's probably the first thing I'd make sure the team understands before using it.

What Bitrise Actually Gives You

There are two main parts to the setup.

1. CodePush inside Bitrise Release Management

This is where you manage deployments, upload releases, and control things like rollout percentages.

2. The SDK inside your application

@bitrise/code-push-sdk

The SDK is responsible for checking whether an update is available and applying it.

So the basic idea is:

Bitrise handles the release side → your app handles the update side.

Once I understood that split, the rest of the setup made a lot more sense.

Before You Touch the Code

You'll need a Bitrise account and a Release Management app.

One thing that caught my attention here is that Release Management and your normal Bitrise CI app aren't necessarily the same thing.

If you're already using Bitrise CI to build your React Native app, Release Management is something you add alongside it.

And if you're releasing for both iOS and Android, you'll need to account for both platforms in your setup.

I'd recommend getting the basic Release Management setup working before touching the React Native project.

It makes troubleshooting much easier.

Step 1: Create a Deployment

Inside the Release Management app, go to:

Configuration → Deployments → New deployment

For a real project, I'd probably start with:

Staging
Production

That separation is useful.

You don't want to be testing an update and accidentally send it to everyone using the production app.

You can either provide an existing deployment key or let Bitrise generate one.

One thing about deployment keys

This initially confused me.

The deployment key looks like something you'd normally hide in a secret.

But it isn't really a password.

The key eventually gets included in the application, which means someone with access to the built app could potentially extract it.

It's better to think of it as an identifier for the deployment rather than a secret credential.

Your API token is the thing you actually need to protect.

Step 2: Install the SDK

The first command is simple:

npm install @bitrise/code-push-sdk

After that, things depend on how your React Native app was created.

And this is one place where I wouldn't recommend blindly copying configuration from a random tutorial.

Expo and bare React Native aren't configured in exactly the same way.

If You're Using Expo

With Expo, you add the CodePush plugin to your app.json.

You'll need your deployment key and the CodePush server URL.

The URL follows this pattern:

https://<workspace-slug>.codepush.bitrise.io

Then run:

npx expo prebuild

I like doing this step because it lets you check whether the configuration actually made its way into the generated native project.

If You're Using Bare React Native

This takes a little more work.

On iOS, you'll need to make a change in AppDelegate.swift so CodePush can resolve the active release bundle instead of always using the default bundled JavaScript.

You'll also configure the deployment key and server URL in Info.plist.

On Android, the configuration goes into strings.xml, along with some Gradle configuration and a change to how MainApplication.kt resolves the JavaScript bundle.

None of these changes are particularly complicated.

They're just easy places to make a small mistake.

That's why I found it better to follow the Bitrise documentation closely rather than trying to recreate the setup from memory.

Step 3: Think About the User Experience

This was probably the part I didn't think about enough at first.

It's easy to look at CodePush as:

"There's an update, download it."

But you actually have choices about how that update reaches the user.

For example, with:

codePush.sync()

you can use different behaviours.

Silent update

The update downloads in the background and is applied when the app restarts.

For the user, the whole thing can be almost invisible.

Update when the app resumes

Instead of checking only when the app starts, the app can check when it comes back into the foreground.

Interactive update

The user gets feedback that an update is available and can be shown the download process.

For mandatory updates, you can also prevent the user from simply dismissing them.

You can even use progress callbacks if you want to show download progress.

I think this is worth deciding before you ship your first update.

The technically easiest option isn't necessarily the best experience for your users.

Step 4: Build and Release

Once the application is configured, you need to create the JavaScript bundle.

For a bare React Native project:

npx react-native bundle \
--platform ios \
--dev false \
--entry-file index.js \
--bundle-output ./build/main.jsbundle \
--assets-dest ./build

For Expo:

expo export:embed

You then package the generated files and upload them to Release Management.

You can do this manually from the dashboard, which is probably the easiest way to test your first release.

Once you've confirmed everything works, that's when I'd start thinking about automation.

Don't Immediately Send It to Everyone

This is one feature I'd definitely use in production.

When you create the release, you can control the rollout.

You can target specific app versions and use version ranges such as:

1.2.x

or:

>=1.2.3 <1.2.7

You can also mark an update as mandatory and control the percentage of users who receive it.

That last part is important.

Imagine you've made a JavaScript change that you think is completely safe.

Instead of immediately sending it to 100% of your users, you can start with a smaller percentage.

If everything looks good, increase the rollout.

That gives you a little more room to catch problems before they become everyone's problem.

What About Automation?

Once you've manually released a couple of updates, you'll probably notice the repetitive part:

Build the bundle.
Zip it.
Upload it.
Do it again for the other platform.

It works.

But I wouldn't want to do that every week.

That's where Bitrise CI comes in.

You can automate the build and upload process so that a workflow handles it for you.

One approach I like is connecting the release to an approved and labelled pull request rather than automatically releasing every merge.

That gives you a deliberate point where someone says:

"This is ready to ship."

The CI system then takes care of the repetitive work.

I wrote separately about that automation process because there are a few more details around credentials, versioning, and workflow configuration.

My Checklist Before the First Release

If I were setting this up on another project, this is the checklist I'd keep nearby:

Bitrise Release Management is configured

iOS and Android are accounted for

Staging and Production deployments are created

@bitrise/code-push-sdk is installed

Expo or native React Native configuration is complete

The app can successfully check for updates

The sync behaviour has been chosen

The first release is tested with a small rollout

Production releases have an automation plan

So, Is CodePush Worth It?

I think it makes the most sense when you regularly have small JavaScript-level fixes that don't justify going through another full app-store release.

It's not a replacement for your normal mobile release pipeline.

Native changes still need a normal build and store submission.

But if your team has ever had to submit a new mobile build because of something that could have been fixed entirely in JavaScript, that's the gap CodePush is trying to close.

For me, the biggest takeaway wasn't just:

"I can push an update without the App Store."

It was realizing that there are several decisions around how that update gets delivered, who receives it, and what the user experiences when it arrives.

That's the part I'd pay attention to before putting CodePush into a production app.

Have You Used CodePush?

Have you used CodePush or another OTA update solution with React Native?

I'd love to hear how you handled it, especially if you ran into problems during setup.

What would you add to this checklist?

Let's not water down the terms Continuous Integration and Continuous Delivery

Mon, 21 Sep 2026 11:16:28 +0000

One of the most perplexing discussions I've had in recent times was about CI/CD. I've earned my CI/CD scout badge at this stage in my career, so the confusion caught me off guard. This post briefly sets out what Continuous Integration (CI) and Continuous Delivery (CD) mean and addresses the most common misconceptions in the discussion.

My point isn't just semantic. These terms carry decades of hard-won engineering practice, and when we let their meaning drift or decay, we lose crucial wisdom.

Continuous Integration

Continuous Integration is well named compared to many concepts in our industry. The term "integration" refers to merging your changes into the main branch in version control, and the term "continuous" means you do it often. When Kent Beck popularized the term in Extreme Programming Explained (1999), he set an upper limit of a day. After 26 years of progress, you should expect to be doing it far more frequently than that.

You might understand the concept better if we look at examples that don't fit the definition. If you have code on your machine that's more than a day old, you're not practicing Continuous Integration. If you have a branch that's more than a day old, you're not practicing Continuous Integration. It doesn't matter how often you merge from main into your branch, because main is completely unaware of changes, like yours, that haven't been merged yet. That means you have diverging software versions, and at some point, you have to make them all converge.

Continuous Delivery

Continuous Delivery is an approach focused on delivering all changes to users quickly, safely, and sustainably. You keep your code in a deployable state at all times and reject the idea of separate integration, testing, and hardening phases. There are specific technical practices involved here because you need a deployment pipeline that reduces risk and increases your confidence that the software version works.

At this stage in our industry's history, almost everything involved in answering the question "Is this deployable?" should be automated as part of the deployment pipeline. A decade ago, perhaps, you could reasonably list a few stages that were economically unviable for automation. The scale and pace of modern software delivery mean this is rarely the case now.

The simple test is to ask, "Is the software deployable?" If you can't answer this question every day, you're not doing Continuous Delivery. In their book on Continuous Delivery (2010), Dave Farley and Jez Humble detail the principles and practices that let you answer this question, and the research has increased our confidence that this is what good software delivery looks like.

Misconceptions

Let's turn our attention to where the conversation left the trail.

CI doesn't just mean automated builds

One of the most common sources of confusion is that CI just means automated builds and tests. While it's difficult to practice Continuous Integration without automation, the crucial property of Continuous Integration is that all changes are in a shared mainline, not on developer machines or in long-lived branches. Having more than 3 branches or keeping branches open for more than a day slows delivery throughput and makes the software less stable, as reinforced by long-running research.

There's a human process at the heart of CI: each developer chooses to commit their changes frequently to main. Without people deciding to do this, there is no CI.

CI doesn't allow feature branching

There's a misconception that you can do CI and choose a branching strategy, like feature branches. Any branching strategy that keeps a change out of main for more than a day cannot be called CI. You may still choose to use feature branches despite the drawbacks, but you have to stop calling it Continuous Integration, as you aren't continuously integrating all changes.

CD doesn't mean Continuous Deployment, though it can

CD means "Continuous Delivery". It means your software is deployable at all times, and you can test this daily by asking, "Can we deploy the software?" If you don't have a way to answer the question confidently, you're not doing Continuous Delivery. You may choose to deploy every validated software version automatically, which is Continuous Deployment, or you may need to be more tactical about when you deploy. Either is fine.

For both Continuous Delivery and Continuous Deployment, you need high levels of automation and sufficient risk coverage within that automation so you know you can deploy. Continuous Delivery should mean you can deploy at the press of a button, and Continuous Deployment removes the requirement to press the button.

You can't do CD without CI

Some folks were confident that you can do Continuous Delivery without doing Continuous Integration, but the literature on this is very clear. Farley and Humble explicitly include Continuous Integration as a required practice. There's a whole chapter dedicated to it (p. 55 of Continuous Delivery) that adequately explains how it should work and why it's not optional.

CI/CD isn't vague or open to interpretation

Another claim that came up more than once was that CI/CD is vague, open to interpretation, or lacks the detail on "how" you should apply the approach. This does a disservice to those who conscientiously captured the approach, whether that's Kent Beck describing Continuous Integration in Extreme Programming Explained (1999) or Farley and Humble detailing both Continuous Integration and Continuous Delivery in their book Continuous Delivery (2010).

Resist the diffusion

There is a constant temptation in the tech industry to allow semantic diffusion to erode our progress. When we package up a cartload of wisdom into a term, we have to resist the diffusion that follows when people oversimplify our term, remove crucial parts of the whole, or try to brand something with the label that pushes us backward.

We have a history of giving up on these battles, as we seem to have done with Agile and DevOps. These terms had an important meaning that represented a significant step forward for our industry, and losing our energy in their defense lets the perpetual "Late Majority" and "Laggards" (from Crossing the Chasm) drag us all back into their old ways.

CI and CD deserve better.

Release Gates Should Be Risk-Based, Not Checklist-Based

Wed, 09 Sep 2026 05:04:03 +0000

A small copy change and a database migration probably shouldn't go through exactly the same release process.

Yet in many teams, they do.

Both need the same approvals. Both follow the same checklist. Both wait for the same people, regardless of what actually changed.

The intention is good: create a consistent process and prevent risky releases.

But consistency doesn't have to mean treating every change as if it carries the same risk.

I think a better question is:

What evidence and controls does this particular release need?

The problem with a fixed checklist

A typical release checklist might include things like:

CI passed
Regression tests passed
Security scan completed
QA approved
Product owner approved
Rollback plan available
Monitoring checked
Deployment window confirmed

None of those are bad checks. Most are useful, and some may be mandatory for every release.

The problem starts when the checklist becomes the decision.

If every box is ticked, the release is considered ready. If one box isn't ticked, the release stops.

That can create two very different problems.

A low-risk change may wait for approvals that don't add meaningful value. Meanwhile, a high-risk change may pass the same checklist without receiving the additional scrutiny it actually needs.

The process is consistent, but the level of control isn't necessarily appropriate.

Two releases, two very different risks

Imagine two release candidates.

Release A

A small copy change on a low-impact page.

The change is isolated. There are no dependency updates, database changes, or changes to authentication. Automated checks pass, and rollback is straightforward.

Release B

An authentication refactor that also includes a database migration and a new external dependency.

The automated checks pass, but the rollback procedure hasn't been exercised recently. The migration affects production data, and the change touches a critical user workflow.

Should both releases require exactly the same evidence?

Probably not.

Release A may be suitable for an automated decision once the required baseline checks are satisfied.

Release B deserves more attention to migration validation, security, dependency behavior, rollback readiness, and possibly an explicit approval from the relevant owner.

The point isn't that Release B must automatically be rejected.

The point is that the risk should determine which additional questions need to be answered.

Start with a baseline, not with zero gates

Risk-based doesn't mean optional quality.

Every team still needs a baseline of controls that applies to all releases.

For example, a team might require a versioned artifact, successful required CI checks, no unresolved blocking security findings, and a valid deployment path.

Those are the minimum conditions for entering the release decision process.

Then, additional controls can be activated based on the change.

Conceptually:

Required baseline
       +
Change-specific risk
       +
Relevant evidence
       =
Release decision

The baseline prevents teams from bypassing essential controls just because a change looks small.

The risk-specific layer prevents every release from being forced through the same expensive process.

Both matter.

What should trigger additional gates?

I would start with a small number of risk dimensions rather than trying to invent a complicated scoring model immediately.

Change impact. How many services, users, or critical workflows could be affected? Is the change isolated, or does it cross system boundaries?

Change type. Does it involve authentication, payments, permissions, infrastructure, database migrations, or a major dependency update?

Evidence quality. Do we have relevant test coverage? Was the migration validated? Are the results current and tied to the actual release candidate?

Operational readiness. Is rollback feasible? Are health checks and monitoring available? Is the team prepared to respond if something goes wrong?

Business context. Is this a routine release, a critical hotfix, or a change during a high-traffic period? What is the cost of delaying it compared with the risk of shipping it?

Not every team needs all of these dimensions. The important thing is to identify the ones that actually influence release risk in your environment.

A simple risk-based gate model

A lightweight implementation could look something like this:

Release context	Additional evidence or control
Small, isolated change	Baseline checks may be sufficient
Database migration	Migration validation, backup and rollback assessment
Authentication or permission change	Relevant security review and targeted regression
Critical dependency update	Compatibility and dependency-impact evidence
High-impact service change	Operational readiness and appropriate owner approval
Reduced test coverage	Explicit risk assessment and compensating controls
Unverified rollback	Release held or exceptional approval under defined policy

This isn't a universal policy. It's an example of how a team could make its controls more relevant to the change.

The important part is that the gates are triggered by meaningful risk, not by the desire to add more process.

Some gates should never be optional

There is an important distinction between a conditional gate and a mandatory control.

A team may decide that certain conditions are non-negotiable.

For example, a critical security vulnerability in a release artifact might be an automatic blocker. A missing required approval for a regulated change might also be a blocker. An artifact that cannot be identified or verified may not be eligible for release at all.

Those rules should not disappear because someone labels the change "low risk."

Risk-based release governance should make mandatory controls explicit.

It should also make exceptions explicit.

If an exception is allowed, the process should define who can authorize it, what evidence is required, how long it remains valid, and what compensating controls are needed.

Otherwise, "risk-based" can quietly become "we skip the checks when we're in a hurry."

Conditional GO is where this becomes interesting

Not every release fits neatly into GO or NO GO.

Sometimes the evidence is sufficient to proceed, but only under specific conditions.

For example:

Decision: CONDITIONAL GO

Reason:
- Required automated checks passed
- No blocking security findings
- Minor observability gap remains

Conditions:
- Enhanced monitoring during rollout
- Named rollback owner available
- Rollout limited to a small user segment
- Production health reviewed before expansion

That's very different from simply ignoring a warning.

The risk is acknowledged. The conditions are explicit. Someone is responsible for ensuring they are met.

And if the conditions aren't satisfied, the release should not proceed under that authorization.

A conditional decision should never be a vague way of saying, "We know there's a problem, but let's ship anyway."

Human approval should add judgment

I'm not against human approval gates.

But I think we need to be careful about what we're asking humans to do.

If every release requires the same person to click Approve, regardless of risk, the approval can become routine.

People get used to seeing green checks. They trust the process. Eventually, the approval becomes another step to get through.

A more useful model is to involve humans when their judgment actually adds value.

For example, when the evidence is incomplete, the business impact is significant, the release includes an unusual change, or the risk exceeds the team's normal policy.

The approver should be able to see the relevant context without manually reconstructing it from five different systems.

That means automation should collect and structure the evidence, while humans resolve the questions that cannot be answered by a simple rule.

Don't confuse a risk score with a decision

It's tempting to turn all of this into a single number.

Release Risk Score: 72

But what does 72 actually mean?

Is it 72 out of 100? Is higher better or worse? Which signals contributed to it? Is a missing rollback plan equivalent to a low test coverage percentage? Can one critical finding be averaged away by ten healthy signals?

A score can be useful as a summary, but it shouldn't hide the underlying evidence.

For example, a release with excellent test coverage and one critical security blocker shouldn't become acceptable just because the average score looks good.

I would rather have a smaller, explainable model than a sophisticated number that nobody can defend.

The decision should be understandable:

Decision: NO GO

Blocking reason:
- Required migration rollback evidence is missing

Supporting context:
- Automated tests passed
- Security checks passed
- Production impact is high

The score, if one exists, should support the explanation — not replace it.

Evidence needs to be current

Another thing that matters is whether the evidence actually belongs to the release being evaluated.

A test report from yesterday may not be relevant if the artifact changed this morning.

A security scan may be outdated after a dependency update.

A staging validation may no longer represent production if the configuration changed.

This is easy to overlook when evidence is scattered across different tools.

A release decision should know which artifact, commit, configuration, or release candidate the evidence refers to.

Otherwise, you can end up with a technically complete checklist built from evidence that no longer describes the thing you're about to ship.

Start with the incidents you've already had

If I were introducing risk-based gates to a small team, I wouldn't begin by designing a huge governance framework.

I'd start with the release problems the team has actually experienced.

Which changes caused incidents?

What evidence was missing?

Which risks were known but not visible to the approver?

What would have helped the team make a better decision?

Maybe the team has had repeated problems with database migrations. Then migration validation and rollback readiness deserve more attention.

Maybe authentication changes have caused regressions. Then targeted security and QA evidence may be appropriate.

Maybe the biggest issue is deployment configuration drift. Then artifact and environment consistency might be the most valuable controls.

The best gates are often the ones that address real failure modes, not the ones that make a process diagram look comprehensive.

The goal is fewer surprises, not more approvals

A good release process shouldn't make every release feel like a compliance exercise.

It should help teams move quickly when the risk is low and slow down when there is a good reason to do so.

That means some releases may need very little additional intervention.

Others may need more evidence, a mitigation plan, or an explicit decision from someone with the right authority.

The process should be predictable, but not blind to context.

And after the release, the team should be able to explain why it was allowed to proceed.

Not just:

"All the boxes were green."

But:

"The required baseline was satisfied, the change-specific risks were evaluated, and the remaining risk was accepted under the appropriate policy."

That's a much stronger answer.

I don't think the future of release governance is a checklist with fifty mandatory steps.

I think it's a smaller set of meaningful controls, activated by the risk of the change, supported by evidence, and connected to clear decision authority.

The goal isn't to make releases harder. It's to make the right releases easy, and the risky ones harder to ship by accident.

This is the fiftth article in my Beyond Green CI series.

How does your team handle release gates today? Are they mostly fixed, or do you adjust the evidence and approvals based on the risk of the change?

A Failed Test Is a Clue, Not a Diagnosis

Wed, 09 Sep 2026 04:57:38 +0000

An end-to-end test fails.

The report is red.

Someone asks, “Is it a bug?”

Maybe.

But at that moment, all we really know is that the test didn't observe the expected behavior.

That distinction sounds small, but it changed the way I investigate failures.

When I started working more deeply with automation, I gradually learned that a failed test is not a diagnosis. It's the beginning of an investigation.

The same failure can have very different causes

Imagine an automated checkout test that expects an order to be created.

The test fails because the order confirmation never appears.

The cause could be an application defect. But it could also be a failed API request, unavailable service, incorrect test data, expired credentials, a deployment issue, an environment configuration problem, or a timing issue in the test.

All of those can produce a similar result in the report.

If we immediately label the failure as a product bug, we may send developers looking in the wrong place.

If we immediately label it as a flaky test, we may hide a real defect.

The first job is to find out which one it is.

Start with the failure, not the assumption

My first step is to understand exactly what failed.

Was it an assertion? A timeout? A missing element? An HTTP error? Did the test fail before the application was even reached?

The details matter.

A test that receives a 500 response is different from a test that cannot connect to the server. A timeout waiting for a button is different from a timeout waiting for an API response.

I also want to know whether the failure is reproducible.

Does it happen every time? Only in CI? Only in one environment? Only when tests run in parallel? Did it start after a particular deployment?

Those questions help narrow the problem before anyone starts changing code.

Check whether the application is actually healthy

One of the most useful habits I developed was to stop looking only at the test report.

If an API test fails, I want to know whether the API is responding outside the test.

If an end-to-end test fails, I want to know whether the underlying service is healthy.

A simple request, a health endpoint, or a look at the service logs can reveal that the test is only reporting a symptom of a larger problem.

For example, a login test might fail because the authentication service is unavailable.

Changing the Playwright locator won't fix that.

Neither will adding another ten seconds to the timeout.

Environment differences matter

I remember how confusing DEV, QA, STG, and PROD were when I first entered QA.

Today, environment differences are one of the first things I consider during an investigation.

The same code can behave differently because of configuration, data, dependencies, feature flags, external integrations, or infrastructure.

If a test passes locally but fails in CI, I want to compare the conditions.

Are the same versions running? Is the test using the same configuration? Are required services available? Is the data equivalent? Are there resource constraints? Is there a difference in browser or runtime versions?

“Works on my machine” isn't a diagnosis either.

It's another clue.

Logs connect the test to the system

This is where my long-standing interest in Linux and Bash became especially useful.

I don't need to understand every internal detail of a service to start investigating it.

But I do need to know how to find relevant information.

Application logs, container logs, system services, network responses, and CI output can help connect the failure in the test to what actually happened in the system.

The important skill isn't memorizing a hundred commands.

It's knowing what question you're trying to answer.

Did the request reach the service? Did the service reject it? Did a dependency fail? Was the process restarted? Was there an error at the same time as the test failure?

When the timestamps and evidence line up, the investigation becomes much more useful.

Don't fix flakiness by hiding it

Flaky tests are frustrating, especially when they slow down a team.

But adding retries or longer waits without understanding the cause can make the situation worse.

Sometimes a retry is a reasonable mitigation. But it should not become a substitute for investigating why the test is unreliable.

Is the application genuinely slow? Is the test waiting for the wrong condition? Is shared test data causing interference? Are tests running in parallel without proper isolation? Is the environment overloaded?

A test that passes on the third attempt may still be telling you something important.

The question is whether you understand what that something is.

Test data deserves more attention than it gets

Test data is another common source of misleading failures.

An account might already exist. A record may have been modified by another test. An external system may have changed state. A cleanup step may not have run.

A test can be technically correct and still fail because its assumptions about data are no longer true.

This is why I increasingly value predictable setup, isolation, cleanup, and clear ownership of test data.

Reliable automation isn't only about good assertions.

It's also about controlling the conditions under which those assertions run.

Look at the change that happened before the failure

When a previously stable test starts failing, I want to know what changed.

A new commit? A dependency update? A deployment? A configuration change? A database migration? A new feature flag?

The recent change isn't automatically the cause, but it provides useful context.

This is also where understanding CI/CD becomes valuable for QA.

A pipeline isn't just a place where tests run. It contains information about the build, dependencies, environment, and changes that produced the result.

The more of that context you can read, the less you have to guess.

A useful investigation produces more than “fixed”

Eventually, the failure is resolved.

Maybe it was an application bug. Maybe the test needed a better wait condition. Maybe a service was unavailable. Maybe the data setup was unreliable.

But I think a good investigation should leave behind something more useful than a green rerun.

What was the actual cause? How did we confirm it? Could the same failure happen again? Can we improve the test, environment, monitoring, or process so it's easier to diagnose next time?

A failure is an opportunity to improve the system's ability to explain itself.

That matters to QA, DevOps, and everyone responsible for reliable delivery.

The broader lesson

The deeper I moved into automation, the more I realized that debugging requires an expanding field of view.

Sometimes the answer is in the test.

Sometimes it's in the application.

Sometimes it's in the environment, CI, infrastructure, or deployment history.

You don't need to be an expert in every layer to investigate effectively.

But you do need to be willing to look beyond the layer where the failure first appeared.

A red test is useful evidence.

It tells us something didn't go as expected.

The real engineering work is finding out why.

This is the second article in my From QA to DevOps / Release Engineering series.

When an automated test fails in your team, what's the first thing you check outside the test report?

Canary Releases: How to Stop Deploying Broken Code to Everyone

Tue, 01 Sep 2026 23:35:23 +0000

What is a Canary Release?

A canary release is a modern software deployment strategy where a newly developed version of an application is rolled out to a tiny fraction of real users before making it available to the entire user base. This technique serves as an invaluable safety mechanism to identify bugs, performance degradations, or architectural errors under real-world conditions without risking a widespread outage. If the update proves stable for this initial cohort, it is progressively rolled out to everyone else; otherwise, it is immediately rolled back with near-zero impact on the general public.

The Canary in the Coal Mine

Historically, this concept stems from a crucial mining practice: coal miners would carry a live canary down into the deep subterranean tunnels because the birds are far more sensitive to toxic, odorless gases than humans are. If the canary stopped singing or collapsed, the miners knew they had to evacuate immediately before they were affected. In the modern tech world, our "canary" is the tiny, targeted group of active users who receive the new code version first. If their software sessions begin throwing errors, crashing, or slowing down, developers receive an immediate alert and can quickly shut down the test, preventing any damage from reaching the larger population.

Why Canary Releases Matter in Daily Operations

In the daily lives of software engineers, canary releases are the difference between a peaceful workday and a high-severity, multi-hour outage. When deploying updates to systems with millions of active users, traditional "big bang" deployments—where 100% of the servers are updated simultaneously—are incredibly dangerous. No matter how much testing is done on offline staging environments, real-world user behavior and high network volumes cannot be perfectly simulated. By routing just 1% to 5% of live traffic to the new update, engineers can closely monitor system metrics like API response times, database query speeds, and error logs. If the canary group encounters an issue, only a tiny sliver of users is affected, saving the business from catastrophic financial and reputational damage.

Code Implementation: Traffic Splitting

Here is a simple Node.js-style routing mechanism that shows how traffic can be split programmatically based on a user's unique ID:

// Deterministic canary router based on user ID hashes
function getAppVersionForUser(userId) {
  const CANARY_PERCENTAGE = 5; // Route 5% of users to canary
  
  // Generate a stable hash value between 0 and 99 based on the user ID
  const userBucket = Math.abs(userId) % 100;
  
  if (userBucket < CANARY_PERCENTAGE) {
    return "v2.0.0-canary"; // The new, experimental version
  }
  
  return "v1.1.0-stable"; // The reliable, existing version
}

// Example usage:
const userA = { id: 104 }; // 104 % 100 = 4 (Canary)
const userB = { id: 250 }; // 250 % 100 = 50 (Stable)

console.log(getAppVersionForUser(userA.id)); // Outputs: v2.0.0-canary
console.log(getAppVersionForUser(userB.id)); // Outputs: v1.1.0-stable

Key Takeaway

Ultimately, canary releases remove the fear and uncertainty from the deployment pipeline. By transforming launches from high-stakes, stressful events into gradual, data-driven experiments, development teams can safely ship new code multiple times a day while maintaining robust, uninterrupted uptime for their global customer base.

Resources

GitHub Repository: react-hook-lab
react-hook-lab: npm package
Connect with me on LinkedIn: Saurav Pandey

Originally published on my blog. You can read the alternative breakdown here.

Who Actually Owns the Release Decision?

Fri, 28 Aug 2026 07:16:54 +0000

The pipeline is green.

QA signed off.

The deployment window is open.

Someone asks:

“Are we good to release?”

And then something interesting happens.

Everyone looks at someone else.

The developer knows the code.

QA knows what was tested.

DevOps knows whether the deployment can run.

Security knows about the findings.

The product owner understands the business impact.

But who actually owns the final decision?

In many teams, the answer is surprisingly unclear.

And I think that ambiguity is becoming a bigger problem as software delivery gets more automated.

Release ownership is often implicit

Most teams have a release process.

Far fewer have a clearly defined release decision process.

There is usually some combination of:

CI checks
automated tests
QA approval
pull request reviews
security scans
deployment jobs
Slack conversations
change tickets
someone saying “looks good to me”

Eventually, the release happens.

But if you ask:

Why was this release considered safe enough to ship?

the answer is often reconstructed afterwards.

Maybe the pipeline was green.

Maybe QA approved it.

Maybe an engineering lead gave a thumbs up.

Maybe nobody saw a reason to stop it.

That last one is more common than we probably like to admit.

“Nobody stopped it” is not a release decision

There is an important difference between:

Nothing blocked the release.

and:

We evaluated the available evidence and decided the release risk was acceptable.

The first is passive.

The second is a decision.

Many delivery systems are designed around the first model.

A release moves forward unless something fails.

Build       PASS
Tests       PASS
Security    PASS
Approval    PASS

            ↓

          DEPLOY

That works well when the important risks can be represented as binary checks.

But releases are rarely that simple.

Imagine this:

Tests                     PASS
Security scan             PASS
Build                     PASS

Database migration        YES
Authentication change     YES
Rollback confidence       MEDIUM
Observability coverage    PARTIAL
Deployment timing         PEAK HOURS

Nothing necessarily failed.

But someone still needs to decide whether those signals add up to an acceptable release.

That is not execution.

That is judgment.

Different people own different pieces of the truth

One reason release ownership becomes fuzzy is that no single person has all the information.

A developer might know:

“The implementation is small and well understood.”

QA might know:

“The regression suite passed, but this integration wasn’t fully covered.”

Security might know:

“There is a finding, but it is low severity and unrelated to this change.”

Operations might know:

“Rollback is technically possible, but it hasn’t been tested recently.”

The product team might know:

“This release needs to go out today because it fixes an important customer issue.”

Each perspective is valid.

None of them alone represents the whole release.

That means the problem isn’t simply finding the right person to press the button.

The real problem is bringing the evidence together before that decision is made.

Approval and decision are not the same thing

We often treat approvals as if they solve this problem.

They help, but an approval is only useful if we understand what is being approved.

Consider a typical approval step:

Production deployment waiting for approval.

Approve / Reject

What exactly does “Approve” mean?

Did the approver review the test results?

Did they see the security findings?

Do they know that the release contains a database migration?

Did they check rollback readiness?

Do they know whether the change affects authentication or payments?

Or are they simply approving because the previous pipeline stages are green?

An approval button does not automatically create a meaningful decision process.

Sometimes it just moves responsibility from automation to a human without giving that human better information.

Humans should make decisions, not collect evidence manually

This becomes especially important when teams introduce human approval gates.

The common pattern is:

Automation runs
      ↓
Human approval
      ↓
Deployment

But the human often has to investigate everything manually.

Open the CI pipeline.

Check the test report.

Look at the pull request.

Check security.

Search Slack.

Ask whether staging was tested.

Find the deployment notes.

Then make a decision.

At that point the human is not really acting as a decision-maker.

They are acting as an evidence aggregator.

And that is exactly the part automation should help with.

A better model might look like this:

CI / QA / Security / Change / Operations
                   ↓
             Evidence layer
                   ↓
              Risk context
                   ↓
             Human decision
                   ↓
                Release

Automation gathers and structures the evidence.

Humans apply judgment where judgment is actually needed.

Not every release needs the same level of control

Another problem with traditional release gates is that they are often static.

Every release follows the same workflow.

But risk is not static.

Compare these two changes.

Release A

Copy change
No dependency updates
No database changes
Full regression passed
Easy rollback

Release B

Authentication refactor
Database migration
New external dependency
Partial regression
Rollback requires data restoration

Should they really require exactly the same release process?

Probably not.

The first may need very little additional review.

The second probably deserves significantly more evidence and possibly explicit approval.

This is why I think release processes should become more risk-aware rather than simply adding more mandatory gates.

The question shouldn’t be:

Did this release complete all 12 steps?

It should be:

Given the risk of this particular change, do we have enough evidence to release it?

A release decision should be explainable

There is another useful test.

After a release, could someone explain why it was approved?

Not:

“Because the pipeline was green.”

But something closer to:

Decision: GO

Why:
- regression suite passed
- no critical security findings
- affected service has healthy staging metrics
- rollback artifact is available
- database migration validated
- required approval received

Or perhaps:

Decision: CONDITIONAL GO

Why:
- automated checks passed
- minor observability gap remains
- release approved with enhanced monitoring
- rollback owner assigned

And sometimes:

Decision: NO GO

Why:
- critical workflow not covered
- rollback path unverified
- production dependency currently unstable

That creates something extremely valuable:

decision history.

Not just what happened.

But why.

Release history usually tells us what happened

Most engineering systems are already very good at recording events.

We can usually find:

which commit was deployed
which pipeline ran
which tests passed
who merged the pull request
when the deployment started
whether the deployment succeeded

That is useful.

But during an incident, audit, or retrospective, another question quickly appears:

Why did we believe this release was safe enough to ship?

That answer is often much harder to reconstruct.

The evidence may still exist, but it is scattered across multiple systems.

A CI run.

A test report.

A security scanner.

A ticket.

A Slack message.

Someone’s memory.

The release happened because all of those signals somehow formed a decision.

But the decision itself was never captured.

That feels like an important gap.

This matters even more with AI-assisted development

Software creation is accelerating.

AI can generate code.

AI can generate tests.

AI can review pull requests.

AI can fix pipeline failures.

AI agents are increasingly capable of performing parts of the delivery workflow themselves.

Eventually, much of the mechanical software delivery process may become highly automated.

That makes one question increasingly important:

What should remain a decision?

The answer probably isn’t:

Everything must be manually approved.

That would destroy much of the value of automation.

But the answer also shouldn’t be:

The pipeline passed, therefore release.

As automation increases, teams will need to become much more explicit about the boundary between:

execution

and

authority.

Machines are excellent at collecting signals, evaluating rules, and enforcing policy.

Humans are still valuable when context, uncertainty, trade-offs, and business risk matter.

The interesting challenge is designing the boundary between the two.

The decision should depend on risk

I don’t think every release needs a human standing in front of a giant approval button.

That would simply create another bottleneck.

Low-risk releases should be able to move quickly.

If the evidence is strong, the change is small, rollback is easy, and the affected systems are well understood, automation may be perfectly capable of approving the release automatically.

But as risk increases, the decision process can change.

For example:

LOW RISK
    ↓
Evidence sufficient
    ↓
Automatic GO

A more complex release might look like:

MEDIUM RISK
     ↓
Evidence sufficient
     ↓
Conditional approval
     ↓
Human review
     ↓
GO

And a high-risk release might produce:

HIGH RISK
    ↓
Missing critical evidence
    ↓
NO GO

The important part is that the decision is based on the release context.

Not simply on whether a static checklist turned green.

Maybe nobody should “own” the release alone

This leads to a slightly different way of thinking about release ownership.

Perhaps the release decision should not belong entirely to one person or one tool.

Instead:

Engineering provides change evidence

QA provides quality evidence

Security provides security evidence

Operations provides deployment evidence

Automation evaluates policy

Humans resolve uncertainty

And the final release decision becomes the result of that system.

Not one person’s intuition.

Not one green pipeline.

Not one approval button.

A decision backed by evidence.

The release button is the easy part

We have become very good at automating deployment.

Running containers, updating infrastructure, promoting artifacts, and rolling out applications can now happen in seconds.

Modern delivery systems are extremely good at answering:

How do we deploy this?

The harder question comes immediately before that:

Should we deploy this?

And if the answer is yes:

Why are we confident enough to do it?

That is a very different problem.

It combines engineering signals, quality evidence, operational context, security information, policy, and sometimes human judgment.

I think that question is going to become one of the most important parts of modern software delivery.

Especially as everything around it becomes increasingly automated.

So maybe the real question isn’t:

Who owns the release button?

It is:

Who — or what — owns the release decision?

And more importantly:

Can that decision be explained afterwards?

How does your team handle this today?

Who actually owns the decision that a release is ready to go?

MLOps for Engineers: How to Take an AI Model from Jupyter to Production Without Dying in the Attempt

Thu, 27 Aug 2026 19:08:27 +0000

There is a statistic from Gartner that every data team knows and most prefer to ignore: 87% of Machine Learning and AI projects never make it to production. They remain trapped in "prototype limbo": a Jupyter notebook on a data scientist's laptop boasting an impressive 96% accuracy over a static CSV file from 2023, but which no one knows how to deploy, version, update, or monitor within an enterprise's live infrastructure.

The reason for this widespread failure is neither mathematical nor algorithmic. It's not a matter of needing more hyperparameters to tune or deeper neural network layers to stack. It is a software engineering and operations failure.

As demonstrated in Google's landmark paper "Hidden Technical Debt in Machine Learning Systems" (Sculley et al.), the actual ML model code accounts for a mere 5% to 10% of the entire production system. The remaining 90% is infrastructure: data extraction and verification, dependency management, automated retraining, artifact versioning, observability against the dreaded Data Drift, and continuous governance.

This integrative discipline is called MLOps (Machine Learning Operations). And after building, deploying, and operating real production pipelines on this blog — from demand forecasting with Prophet in the S&OP series to autonomous agents in the Autopilot series and the Obsolescence Radar —, this article is the practical guide that distills the journey from a sandbox script to an industrial production system.

Anatomy of the Problem: Why Traditional Software Engineering Falls Short with ML

In traditional software development (DevOps), system behavior depends entirely on source code. If you write a deterministic function, test it with unit tests, and deploy it via CI/CD, the system behaves predictably as long as the underlying infrastructure remains healthy.

In Machine Learning and generative AI systems, behavior depends upon an interdependent trinity: Code + Data + Model.

Code may remain unchanged, yet if the statistical distribution of real-world data shifts (which happens constantly across supply chains, financial markets, and customer behaviors), model performance degrades silently.
Reproducibility is non-trivial: retraining the exact same Python script with today's data produces a completely different binary artifact than last week's run.
Failures don't throw an HTTP 500 error: a production model doesn't crash like a web server; it simply begins serving garbage predictions with 99% statistical confidence.

To keep your system from becoming an uncontrollable black box, your architecture must stand upon four fundamental pillars.

Pillar 1: Comprehensive Versioning (Code, Data, and Models)

If you cannot recreate the exact environment, data snapshot, and parameters that produced a prediction six months ago, your system is neither reproducible nor auditable. In regulated enterprise environments, this is not merely good engineering hygiene; it is a strict legal requirement under the EU AI Act (Article 12 on automated logging and traceability).

Versioning in MLOps spans three distinct layers:

Code Versioning: Standard Git. Centralized repository with protected branches, pull request reviews, and version tags.
Data Versioning (DVC / Delta Lake): Git was never designed to store gigabyte- or terabyte-scale binary files. Tools like DVC (Data Version Control) create lightweight metadata pointer files versioned in Git, while actual datasets reside in object storage (Amazon S3, Google Cloud Storage, or Supabase Storage). This allows a simple git checkout v1.2.0 to restore both the training code and the exact data snapshot that fed it.
Model Registry (MLflow / Weights & Biases): A centralized catalog acting as the "Docker Hub" for trained models. Each registered model tracks serialized weights (ONNX, Pickle, Safetensors), hyperparameters, validation metrics, author, associated Git commit hash, and lifecycle stage (Staging, Production, Archived).

Pillar 2: Experiment Tracking and Pipeline Automation

A data scientist's or ML engineer's workflow is inherently iterative. Testing dozens of feature permutations, model algorithms, Pandas transformations, and regularization parameters without systematic logging leads straight to chaos: scattered notebook files, models named final_model_v2_really_final.pkl, and complete loss of institutional knowledge.

MLflow and Weights & Biases (W&B) solve this by systematically intercepting every training run:

import mlflow
import mlflow.prophet
from prophet import Prophet

# Start experiment tracking
mlflow.set_experiment("sop_demand_forecasting")

with mlflow.start_run(run_name="prophet_multiplicative_v3"):
    # Log configuration parameters
    params = {
        "seasonality_mode": "multiplicative",
        "changepoint_prior_scale": 0.05,
        "n_changepoints": 20
    }
    mlflow.log_params(params)
    
    # Train the model
    model = Prophet(**params)
    model.fit(train_df)
    
    # Evaluate and log performance metrics
    metrics = evaluate_forecast(model, test_df)
    mlflow.log_metrics({
        "mape": metrics["mape"],
        "rmse": metrics["rmse"],
        "coverage_p95": metrics["coverage"]
    })
    
    # Automatically register the artifact
    mlflow.prophet.log_model(model, artifact_path="prophet_model")

By structuring experiments with automated tracking, comparing 50 model iterations transforms from a guessing game into an instant analytical query on a unified dashboard.

Pillar 3: CI/CD and Automated Deployment (CT: Continuous Training)

In MLOps, CI/CD expands to incorporate a vital third dimension: Continuous Training (CT).

CI (Continuous Integration): Goes beyond validating Python syntax and running unit tests (pytest, flake8). It executes specialized data validation tests: verifying that incoming schemas match expectations, null values remain within acceptable tolerances, and feature ranges stay valid (using libraries like Great Expectations or Pydantic).
CD (Continuous Delivery / Deployment): Packages the validated model into an optimized Docker container or a FastAPI microservice (as showcased in Observability Part 6) and automatically deploys it upon passing regression test suites.
CT (Continuous Training): When production monitoring detects performance degradation or new data batches arrive, an automated pipeline (orchestrated via GitHub Actions, Prefect, or Airflow) retrains the model, verifies that new metrics surpass the baseline champion model, and automatically promotes the candidate to the Model Registry.

Pillar 4: Production Observability (The War Against Drift)

Once your model is deployed and serving batch or real-time inferences, the real challenge begins. Models degrade over time due to two distinct mathematical phenomena:

1. Data Drift (Covariate Shift)

The statistical distribution of input variables ($P(X)$) changes relative to the training distribution, even if the underlying relationship between inputs and outputs holds steady.

Real-world example: A demand forecasting model trained prior to a global supply chain disruption experiences a sudden surge in supplier lead times. The model receives valid numbers, but operates in a vector space region where it was never trained.

2. Concept Drift

The mathematical relationship between input features and target variables ($P(Y|X)$) shifts over time.

Real-world example: In customer purchasing predictions, a product historically bought primarily during winter becomes an all-season hit due to a social media trend. The inputs remain identical, but consumer behavior has fundamentally evolved.

# Data Drift detection example using Evidently AI
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

data_drift_report = Report(metrics=[DataDriftPreset()])
data_drift_report.run(reference_data=reference_df, current_data=production_df)

# If drift exceeds threshold, trigger automatic alerting & retraining
if data_drift_report.as_dict()["metrics"][0]["result"]["dataset_drift"]:
    trigger_mlops_alert("DATA_DRIFT_DETECTED: Triggering automated retraining pipeline")

Tools like Evidently AI or Prometheus + Grafana allow data engineering teams to track statistical drift tests (such as Kolmogorov-Smirnov for numerical features or Chi-Square tests for categorical data) and alert engineers long before degraded accuracy causes financial losses.

Real Production Case: Moving Prophet from Notebook to S&OP Pipeline

To see how MLOps applies in real industrial contexts, consider the evolution of our demand planning pipeline in the S&OP series:

Stage	Initial Sandbox (Jupyter)	Production Architecture (MLOps)
Data Ingestion	Manual loading of static `sales_2023.csv`	Automated Supabase PostgreSQL sync with schema assertion
Data Hygiene	Manual `df.dropna()` without validation	Automated Z-Score outlier detection as detailed in Data Hygiene
Training	Manual top-to-bottom cell execution	Decoupled pipeline running weekly via GitHub Actions
Bayesian Inference	Single point forecast	Probabilistic confidence intervals directly driving Safety Stock calculations
Consumption	Static `plt.show()` inline charts	FastAPI microservice feeding the PuLP linear optimization engine (Part 3)
Monitoring	None	Continuous weekly MAPE tracking triggering automated retraining

This engineering transition transformed an isolated data analysis script into an automated, resilient, enterprise-grade system.

From MLOps to LLMOps: The Agentic Frontier

In 2026, the rise of LLMs and autonomous agent architectures hasn't eliminated the need for MLOps; it has evolved it into LLMOps.

When designing multi-agent workflows like those in the Autopilot series or navigating Fine-Tuning vs Prompt Engineering vs RAG, operational challenges take on fresh complexities:

Automated Prompt CI/CD Regression Testing: Adjusting an agent's system prompt can fix one edge case while subtly breaking three others. LLMOps pipelines run automated regression test suites against curated evaluation datasets using frameworks like RAGAS or DeepEval.
Token Cost and Latency Observability: As discussed in The Hidden Economics of AI, token consumption and inference latency are mission-critical operational metrics that must be monitored just as closely as server CPU or memory utilization.
Proactive Security Governance: Given vulnerabilities like Prompt Injection, LLMOps observability must continuously audit tool execution payloads (Tool Calling / MCP) and flag anomalous behaviors before they trigger unauthorized downstream actions.

The EU AI Act Connection: Compliance as Code

The EU AI Act has elevated MLOps from a best-practice engineering recommendation to a mandatory legal requirement:

Article 9 (Risk Management System): Mandates continuous risk identification and mitigation across the entire AI lifecycle (fulfilled via automated CI/CD and continuous testing).
Article 12 (Record-Keeping & Logging): Requires automated event logging throughout production operations (fulfilled via Model Registries and observability tracking).
Article 15 (Accuracy, Robustness & Cybersecurity): Demands that high-risk systems maintain consistent accuracy benchmarks and resist adversarial manipulation.

Implementing rigorous MLOps practices allows companies to satisfy these regulatory mandates natively through infrastructure-as-code, eliminating the friction and overhead of retrospective audits.

Conclusion: MLOps is the Maturity of Artificial Intelligence

Anyone can clone a repository, open a Jupyter notebook, and fit a model in three lines of Python. But real engineering is not about making an algorithm work once in a sanitized sandbox; it is about ensuring it runs reliably, accurately, and securely ten thousand times a day in production.

The continuous MLOps loop — design, train, validate, deploy, monitor, and retrain — is the purest technological expression of the PDCA (Plan-Do-Check-Adjust) cycle that W. Edwards Deming pioneered for industrial quality excellence.

If you aspire to build AI systems that generate enduring enterprise value, leave behind the comfort of the solitary notebook and embrace the discipline of MLOps. Your future self — and your on-call engineering team — will thank you forever.

Sources of Interest:

Your Tests Passed. That Still Doesn’t Mean You Should Release.

Mon, 24 Aug 2026 15:37:01 +0000

Your Tests Passed. That Still Doesn’t Mean You Should Release.

A green test suite feels reassuring.

Unit tests pass. Integration tests pass. End-to-end tests pass. CI is green.

At that point, releasing can feel like the obvious next step.

But there is a problem:

Passing tests prove that the checks you wrote passed. They do not prove that the release is a good decision.

That distinction matters more as systems become larger, deployments become more frequent, and release pipelines become increasingly automated.

Tests answer a narrow question

A test tells you whether a specific expectation was satisfied.

A unit test might tell you that a function returned the expected value.

An API test might confirm that an endpoint returned 200.

An end-to-end test might prove that a customer can complete checkout.

These are valuable signals.

But every test is built around assumptions.

Someone decided:

what should be tested,
what data should be used,
which environments matter,
which failure modes are important,
and what should be ignored.

A passing test therefore means:

Under the conditions we tested, the expected behavior occurred.

It does not automatically mean:

This version is safe to release.

Those are two different statements.

A release contains more evidence than test results

Imagine a deployment where every automated test passes.

At the same time:

A dependency was upgraded.

A database migration touches a large production table.

Error rates increased slightly during staging.

A feature flag configuration changed.

The deployment modifies authentication logic.

Observability coverage for the new code is incomplete.

None of these conditions necessarily cause a test failure.

Yet all of them may affect the release decision.

This is where software teams often confuse verification with decision-making.

Tests verify behavior.

Release decisions evaluate risk.

The dangerous assumption behind green pipelines

Many delivery workflows implicitly implement a rule like this:

tests pass
+
build succeeds
+
deployment artifact exists
=
release

That works surprisingly well when systems are small and changes are simple.

But as complexity grows, the equation becomes weaker.

A deployment pipeline might be perfectly healthy while the release itself is questionable.

For example, CI might confirm that:

lint          PASS
unit tests    PASS
integration   PASS
build         PASS
security scan PASS

Yet the release could still involve a risky combination:

large migration
+
critical service
+
low test coverage in changed area
+
no recent production rollback test
+
Friday evening deployment

Every individual check can be green while the combined release risk is still high.

Tests are signals, not decisions

A better model is to think of tests as one category of evidence.

Other evidence might include change size, affected components, dependency changes, migration risk, security findings, incident history, deployment timing, observability readiness, rollback capability, environment differences, or required approvals.

The release decision then becomes something closer to:

Release Decision =
    Test Evidence
  + Change Evidence
  + Operational Evidence
  + Security Evidence
  + Deployment Evidence
  + Organizational Context

Not every project needs all of these.

But the important shift is conceptual:

A test result is evidence presented to a release decision. It is not the release decision itself.

Why this becomes important with AI-generated code

This distinction becomes even more important as AI coding tools become part of normal development.

AI can generate code quickly.

It can generate tests quickly too.

That increases development speed, but it can also create an interesting failure mode:

More code.

More tests.

More green checks.

But not necessarily more understanding of the release risk.

An AI-generated implementation may technically satisfy the tests while introducing architectural changes, new dependencies, unexpected operational behavior, or assumptions that were never encoded into the test suite.

The answer cannot simply be “add more tests.”

Tests are still essential.

But teams also need a layer that asks:

What changed, what evidence do we have, and is that evidence strong enough to release?

The missing layer in many CI/CD systems

Most pipelines are very good at execution.

They can run thousands of tests, build containers, scan dependencies, generate artifacts, and deploy infrastructure.

But pipelines are often much weaker at explaining the final decision.

Why was this release allowed?

Which evidence mattered?

Was anything ignored?

Did the change touch a high-risk component?

Was an exception approved?

What would have caused the release to be blocked?

In many teams, those answers exist only in someone’s head, a Slack conversation, or a manual checklist.

That becomes difficult to scale.

Especially when multiple repositories, services, teams, and automated agents are involved.

From pipeline gates to release evidence

A stronger release process does not replace CI.

It builds on top of it.

CI remains responsible for producing reliable technical signals.

The release layer then evaluates those signals in context.

Instead of:

All checks passed → deploy

the process becomes:

Collect evidence
      ↓
Evaluate release risk
      ↓
Apply policy
      ↓
Require approval when necessary
      ↓
Release / hold / reject

That creates room for more nuanced outcomes than a simple green or red pipeline.

A release might be technically valid but require human approval.

Another might be blocked because rollback capability is missing.

A low-risk change might be automatically approved.

The important part is that the decision is explicit.

Green should mean something

There is nothing wrong with a green CI pipeline.

Green builds are valuable.

Passing tests are valuable.

Fast feedback is valuable.

The problem begins when we assign them more meaning than they actually provide.

A green pipeline should tell us:

The checks we defined passed.

A release decision should tell us:

Given the available evidence and our risk policy, we are willing to ship this change.

Those two statements are related.

They are not identical.

And as software delivery becomes increasingly automated, separating them may become one of the most important design decisions in modern CI/CD systems.

What does “ready to release” mean in your team today? Is it simply a green pipeline, or is there another decision layer after CI?

CI Is Green. What Could Still Go Wrong?

Mon, 24 Aug 2026 15:30:17 +0000

A green CI pipeline is a nice feeling.

The build passed.
Tests are green.
Linting is clean.
Nothing obvious is on fire.

For a few seconds, it feels like the release question has already been answered.

It hasn't.

A green pipeline tells you something useful: the checks you configured have passed under the conditions in which they were executed.

That is not the same thing as saying the release is safe.

And the gap between those two ideas is where a lot of production problems begin.

I touched on this in my previous article about why a green CI pipeline can create a false sense of security. But there is another question that deserves more attention:

If CI is green, what can still go wrong?

Quite a lot, actually.

CI only knows what you taught it to check

This is probably the most important limitation.

CI is not intelligent by default.

It does not understand your product, your users, your business risk, or the weird operational history of your system.

It runs instructions.

If those instructions cover the right risks, great.

If they don't, the pipeline can be perfectly green while the release is still dangerous.

This is not really a failure of CI.

It is a failure of interpretation.

We often treat CI as if it has answered a bigger question than it actually has.

The pipeline says:

"The checks I know about passed."

Humans often hear:

"This release is safe."

Those are very different statements.

The code can be correct and the release can still fail

Imagine a change that passes every unit test and every integration test.

The code itself is fine.

Then it reaches production and fails because of configuration.

Wrong environment variable.

Missing secret.

Different feature flag state.

A service points to the wrong endpoint.

A timeout is configured differently in production.

A permission exists in staging but not in production.

None of those problems necessarily mean your tests were bad.

They mean the release depended on conditions outside the scope of those tests.

This is one of the reasons I think we should stop treating release readiness as purely a testing problem.

Testing is part of it.

It is not the whole thing.

Database migrations are a good example

Database changes are one of those areas where "green" can be misleading.

The migration itself may be syntactically correct.

Tests may pass against a fresh database.

Everything can look completely normal.

But production has something your test environment often does not have:

history.

Real data.

Old records.

Unexpected null values.

Large tables.

Long-running transactions.

Different indexes.

Different traffic patterns.

A migration that takes two seconds in CI can behave very differently against a production dataset that has existed for six years.

The pipeline can be green because nothing it executed was technically wrong.

The release can still be risky because the environment changed the meaning of that change.

Dependencies can change the picture

Modern software rarely runs in isolation.

A release can depend on:

another internal service
a third-party API
a package update
a cloud service
a message queue
a payment provider
a database
an authentication system

Your own repository may be completely healthy.

The dependency chain around it may not be.

Maybe another service has not deployed the expected API yet.

Maybe a backwards-compatibility assumption is wrong.

Maybe one dependency is already degraded.

Maybe the release introduces a combination of versions that nobody has actually tested together.

Again, CI may remain perfectly green.

It tested what it could see.

The release depends on more than that.

Passing tests do not mean the right tests ran

This one is uncomfortable, because it sounds obvious once you say it.

A test suite can have a 100% pass rate while still telling you very little about the change you are about to release.

The interesting question is not:

How many tests passed?

It is:

Did we test the things that matter for this release?

Those are not equivalent.

If somebody changes a critical authentication flow and your pipeline runs 3,000 tests mostly covering catalog functionality, the number "3,000 passed" looks impressive.

But what does it actually tell you about authentication risk?

Not much.

This is why I find the concept of risk coverage more useful than simply looking at test coverage or test count.

What changed?

What can that change affect?

Which checks give us evidence about those areas?

That context matters more than a big green number.

Feature flags do not magically remove risk

Feature flags are useful.

Very useful.

But sometimes they become another source of false confidence.

The thinking becomes:

"It's behind a flag, so the release is safe."

Maybe.

But the code still shipped.

The flag configuration itself can be wrong.

The disabled path may affect initialization.

The new code may introduce dependencies before the flag is evaluated.

The wrong audience may receive the feature.

The rollback path may not behave the way you expect.

Feature flags reduce certain kinds of risk.

They do not eliminate release risk.

That distinction matters.

Rollback is not a guarantee either

Another sentence I hear in software teams is some version of:

"If something goes wrong, we'll just roll it back."

That can be a perfectly reasonable risk strategy.

But "rollback exists" and "rollback is safe" are not the same thing.

What happens if the release includes a destructive migration?

What happens if new data has already been written in a new format?

What happens if another service starts depending on the new behavior?

What happens if the release changes an external contract?

Rollback sounds simple when we imagine software as a single binary moving backwards.

Real systems are messier.

Sometimes the release creates state that cannot be cleanly reversed.

A green CI pipeline usually does not tell you whether your rollback strategy actually works.

Production traffic is different

There is also a class of failures that only appears under real usage.

Concurrency.

Traffic spikes.

Unusual request sequences.

Large payloads.

Rare account states.

Slow upstream services.

Network instability.

Real users are extremely good at discovering combinations that test environments never produce.

This is not an argument against automation.

It is an argument for understanding what automation can and cannot prove.

A test suite can give you evidence.

It cannot give you certainty.

Observability matters before the release, not only after it

One thing I think teams often underestimate is operational readiness.

Suppose the release does contain a problem.

Will you notice?

How quickly?

Do you have the right metrics?

Do you have logs around the changed area?

Will an alert actually fire?

Can you distinguish expected noise from a real regression?

A release with some uncertainty but excellent observability can sometimes be safer than a release with more passing tests but poor visibility.

Because release safety is not only about preventing every possible failure.

It is also about limiting the damage when a failure happens.

That changes the release decision.

Human context still matters

Not every release risk is technical.

Sometimes the biggest risk is simply that nobody has the full picture.

QA knows about one unresolved issue.

A developer knows the fix was rushed.

Operations knows a dependent system has been unstable all afternoon.

Product knows a feature absolutely must not reach one customer segment yet.

Security raised a concern that is technically "non-blocking."

Each piece of information exists.

But it exists in different places, owned by different people.

Then the pipeline turns green.

And everyone assumes somebody else must have checked the rest.

This is where release processes become dangerous.

Not because there was no information.

Because the information never became part of the decision.

So what does green CI actually mean?

I still want green CI.

Obviously.

A red pipeline is useful information too, but I would much rather start from green.

The problem is not the green status.

The problem is giving that status more authority than it deserves.

For me, a green pipeline should mean something closer to:

The automated checks we currently require did not detect a blocking problem.

That is a strong signal.

It is not yet a release decision.

A real release decision needs more context.

Depending on the system, I would want to know things like:

What changed?
How risky is that area?
Which relevant tests actually ran?
Are there unresolved defects?
Are dependencies healthy and compatible?
Are migrations involved?
Are security checks clean?
Do we have sufficient observability?
Can we roll back safely?
Is there any known operational or business risk?

Only then does the question become interesting:

Do we have enough evidence to release?

Maybe the problem is the way we frame CI

We built CI to automate verification.

Over time, many teams started using it as a proxy for release readiness.

That shortcut works surprisingly often.

Which is exactly why it is dangerous.

Most releases with a green pipeline are probably fine.

Then one isn't.

And when you investigate what happened, the answer is often not:

"CI failed."

It is:

"CI never knew that this was something it needed to evaluate."

That is the gap I keep coming back to.

The future of release engineering, in my opinion, is not just more tests and faster pipelines.

It is better decision-making.

More context.

Better evidence.

A clearer understanding of what we know before we put a change in front of users.

Green CI should absolutely make us more confident.

It just shouldn't make us stop asking questions.

Continuous Delivery Office Hours Ep.5: Delivering database changes

Mon, 24 Aug 2026 12:57:43 +0000

In the previous episode, we talked about different service design approaches. This time, we dive deeper into database changes.

Even for teams that can effortlessly deploy their application code, database changes can be more stressful. Changing a schema is a high-stakes operation, and there are many ways to do it badly.

Read on to find out why database deployments are different from application deployments, and what techniques you can use to make them worry-free.

Watch the episode

You can watch the episode below, or read on to find some of the key discussion points.

Watch Continuous Delivery Office Hours Ep.5

Why are databases different

When you change your application's code and discover an issue, it's trivial to revert to the previous version. In rare cases where a change affects your data, there may be cleanup to do.

You can use timestamps and a one-off process to fix those rare data-mess-up instances, but they hint at something fundamentally different about database updates. They have different levels and types of risk associated with them.

If you think backups will save you from a bad database deployment, you haven't yet tried it. Sure, they prevent total data loss, but between your backup and your fix, the data moved. Often by a lot.

There are techniques to apply transactions from the backup up to the point of failure, but if your change caused an issue you needed to roll back, you probably don't want to apply those transactions automatically. Welcome to the "data remediation project".

If you added a new column or table as part of your database change and you need to roll back, you have to decide what to do with any data in those tables. Do you forget it, or do you need to keep hold of it and reapply it later when you make a new attempt to extend the schema?

Modern software teams prefer fix-forward for application issues, which are reasonably easy to roll back. Databases take rollbacks to another level.

Crucial modernization steps

Imagine you met a friend for lunch and they told you they store they application's source code on a network share, rather than in version control. You'd think it was a joke, and when you realize it's not, you'd form a strong opinion about the kind of sloppy outfit they must be running.

When we meet for lunch, what are you going to tell me about your database schema and static data? Please tell me it's all in version control, not on a network share.

You should make all database changes by updating the files in version control and deploying them like you would your application code. You progress the change through environments to ensure it works, and you avoid embarrassment caused by the application failing because someone forgot to add the new column in production.

There are further choices to make, which we'll cover next, but failing to version-control your database is unforgivable.

State-based vs migration-based approaches

On to the first choice for your database project. Do you make state-based or migration-based updates?

State-based schemas describe the desired state of the database. It will list each table with its columns, indexes, and relationships. You use a model-based tool to deploy the database, which compares your current state with the desired state and applies the changes for you.

Some state-based tools convert the differences into standard database scripts, like ALTER TABLE... scripts. Others perform migrations by creating a new table with the changes and moving the data into it. This is important if you're using a technology like replication, which prevents the model migration mechanism from working.

The alternative to state-based database updates is migration-based updates, where you write your own ALTER TABLE scripts. You keep all your scripts in version control and use a tool that applies them in order and tracks when each was applied, preventing the same script from being applied twice to the same database.

The main difference between the two is procedural. You can code-review the migration scripts on demand, but you'll need to review state-based migration scripts after your tools generate the implementation plan, which can make the review task more of a large batch.

Tooling and automation

Whichever approach you use, tooling helps it work. Many of our customers use Redgate tools as part of their deployment process to manage database schema changes, and there's value in leaning on a tool written by folks who care deeply about the problem and how to solve it.

Crucially, you shouldn't make any database changes outside of tool-based automation.

Test data management

Once you've automated your database schema and static data, it's worth considering your test data. When automated or manual acceptance tests fail, it's usually because someone unwittingly messed up the data. Prior test runs often leave data in an inconsistent state, especially if a test failure halts the run.

You can resolve this issue by automating your test data setup. Not only does this make it easy to reset the data during your build and test cycle, but it also lets you provide a self-service runbook for the test team to reset the data in their test environment whenever they need to.

There's an up-front investment in this, but I can promise that it takes fewer hours than fixing your test data a few times.

Database refactoring patterns

The final thought to ponder concerns the steps you take in changing your database schema. When you're in the habit of deploying your database and application in the same release process, you start to depend on this change coordination.

You delete a column from the database and immediately deploy the application, with all references to the deleted column removed. It looks like it works smoothly, but it's a trap.

Imagine you had a critical bug in the application and had to redeploy the previous version. Now you can't, because the previous version will try to read from a column that doesn't exist. You no longer have a quick, easy back out plan, since you have to re-add the column, and you'll also need data to put in it.

This approach also prevents you from making seamless deployments, as even if you progressively roll out the application version, the old version will error out due to the database change. You may deploy in the opposite order, application first, then database. You'll discover the same problem with new columns that the latest version expects to find in the database.

You need to decouple database and application deployments, and there's a whole book on the topic, called Refactoring Databases (Ambler, Sadalage). You can start by following the expand/contract pattern, which splits updates into steps. The principle is that you don't delete a column until the production application has no reads or writes. You add a column and don't reference it in your code until you deploy to production.

This means you can run the current version and the new version of the software against the same database, which means you can progressively roll out the new version and redeploy the prior version without touching the database.

Databases are only as hard as you make them

The database is high-risk, which is why updating it can be scary. I hope you've found this post full of practical advice for making database deployments robust and stress-free.

Database deployments, like application deployments, should be a happy time. You should be celebrating the new features and enhancements you've delivered to your users, not biting your nails and worrying that something's about to go horribly wrong.

Happy deployments!

Continuous Delivery Office Hours is a series of conversations about software delivery, with Tony Kelly, Bob Walker, and Steve Fenton.

You can find more episodes on YouTube, Apple Podcasts, and Pocket Casts.

Your CI Pipeline Is the Most Privileged Machine You Own. You Have Never Scanned It.

Fri, 21 Aug 2026 20:17:57 +0000

You scan your application code. You scan your dependencies. You scan your containers.

You have never scanned the thing that has access to all of them.

Your CI pipeline holds your deploy keys. Your registry credentials. Your cloud tokens. Your signing secrets. It runs on every push, it runs third-party code you did not write, and it runs with more privilege than any developer on your team.

It is the most privileged machine you own. And for most projects, it is completely unreviewed.

Why Attackers Target The Pipeline

An attacker who compromises your laptop gets your laptop.

An attacker who compromises your pipeline gets production.

They do not need to find a vulnerability in your code. They do not need to phish a developer. They need one workflow file that trusts something it should not. The pipeline then does the work for them, with credentials you handed it, inside a network you allowed.

This is not theoretical. Supply chain attacks moved to CI years ago because that is where the credentials live. The Codecov bash uploader compromise in 2021 exposed secrets from thousands of CI environments. In March 2025, the widely used tj-actions/changed-files action was compromised and began leaking secrets into build logs of every repository that referenced it by tag.

Both attacks worked the same way. Trusted third-party code, running with pipeline privileges, on somebody else's push.

1. Actions Pinned To Tags Instead Of Commits

The mistake: Your workflow says uses: some-org/some-action@v3.

A tag is a pointer. Whoever controls the repository controls where it points. If that account is compromised, v3 silently becomes attacker code on your next run. You changed nothing. You approved nothing. You pulled it anyway.

The fix: Pin every third-party action to a full commit SHA.

# Do not do this
uses: some-org/some-action@v3

# Do this
uses: some-org/some-action@a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0

A SHA cannot be moved. Add a comment with the version so future you knows what it was.

How to catch it: grep -rn "uses:.*@v" .github/workflows/ lists every unpinned action you have. Do it now. The result is usually worse than expected.

2. Secrets Available To Every Job

The mistake: Your secrets are repository-wide. Every workflow can read them. That includes the lint job. That includes the job that runs a third-party formatter on a fork's pull request.

A secret is exposed to everything that can read it, not just to the step you were thinking about when you added it.

The fix: Scope secrets to environments, not to the repository, and require approval for the environments that hold production credentials. A test job does not need a deploy key. Give it nothing.

How to catch it: List your repository secrets. For each one, name the single job that needs it. Anything you cannot assign to a specific job is over-shared.

3. Workflows That Run On Untrusted Pull Requests

The mistake: You use pull_request_target because you needed a token that pull_request does not have, and then you check out the pull request head.

That combination runs attacker-controlled code with your secrets in scope. It is the single most reliable way to leak credentials from a public repository.

The fix: Use pull_request for anything that touches untrusted code. If you genuinely need privileged work on a fork's contribution, split it: an unprivileged job builds and uploads an artifact, a separate privileged job consumes the artifact without ever checking out the fork's code.

How to catch it: grep -rn "pull_request_target" .github/workflows/. Every hit needs a written justification. If nobody remembers why it is there, it should not be there.

4. Default Write Permissions On The Job Token

The mistake: You never set permissions:, so the automatic token gets whatever the repository default is. On older repositories, that default is write. Any compromised step in that job can push commits, edit releases, or open pull requests as your CI.

The fix: Default to nothing and grant upward per job.

permissions:
  contents: read

jobs:
  build:
    permissions:
      contents: read
  release:
    permissions:
      contents: write

How to catch it: Any workflow file with no permissions: block is running on defaults. Search for the files that lack it, not the ones that have it.

5. Secrets Printed Into Logs

The mistake: Someone added set -x while debugging, or a step echoes a config object, or a failing command dumps its environment. Masking catches exact string matches. It does not catch a secret that has been base64 encoded, JSON wrapped, or split across a variable expansion.

Build logs on public repositories are public. Artifacts are downloadable.

The fix: Never echo configuration. Remove debug tracing before merge. Treat log output as published, because it is.

How to catch it: Run a secret scanner across your own build logs and artifacts, not just your source tree. Gitleaks and TruffleHog work on any text. Most people never point them at the one place secrets actually get printed.

The Ten Minute Audit

You do not need a new tool for this. Open a terminal in your repository:

# Unpinned third-party actions
grep -rn "uses:.*@v" .github/workflows/

# Workflows that run untrusted code with privileges
grep -rn "pull_request_target" .github/workflows/

# Workflows with no explicit permissions block
grep -rLn "permissions:" .github/workflows/*.yml

# Debug tracing left behind
grep -rn "set -x" .github/workflows/

Four commands. Ten minutes. Fix the SHAs first, then the permissions blocks. That covers most of the realistic attack surface.

Why This Stays Broken

Pipeline security falls between two teams. Application security scanners read your source code, not your workflow files. Infrastructure scanners read your Terraform, not your build steps. The pipeline sits in the gap, and the gap is where the credentials are.

The tools that do read workflow files exist. Checkov covers CI configuration, Semgrep has rules for GitHub Actions, and actionlint catches the structural problems. None of them is the default in a new repository, which is exactly why this keeps working for attackers.

The Bottom Line

Your pipeline has more privilege than you do. Scan it like it does.

Pin your actions to SHAs. Scope your secrets. Never run untrusted code with credentials in reach. Then go look at what your build logs have been printing this whole time.

Discussion Question

Go run grep -rn "uses:.*@v" .github/workflows/ right now. How many unpinned actions did you find? I want to know if anyone is actually at zero.

How do I know when my website is ready for deployment?

Fri, 21 Aug 2026 06:29:41 +0000

Forms passing in staging does not mean the site is fast enough to publish. A short checklist for speed budgets on priority URLs, CI checks, production previews, and what to measure after deploy.

On Thursday the team closed the last functional ticket. Forms submitted, redirects resolved, analytics fired in staging. On Friday someone ran PageSpeed Insights on the production preview URL and mobile INP on checkout was amber while the homepage looked fine.

That sequence is common. Functional QA answers whether the site works. Speed readiness answers whether the build you are about to publish keeps Core Web Vitals inside the budgets you set for the URLs that handle signups, sales, or support traffic. Below is the checklist we use when someone asks how to know a website is ready for deployment, with speed checks listed separately from forms and content review.

Why does passing functional QA not mean the site is ready to deploy?

Functional QA checks behaviour: links resolve, auth flows complete, carts total correctly, CMS entries render, consent banners appear. Speed readiness checks load time and stability on representative page templates under realistic throttling, usually mobile first.

A site can pass the first checklist and still ship a regression. Typical causes include a hero image without dimensions on the homepage template only, a new marketing tag that blocks the main thread on checkout, a font swap that shifts layout on product pages, or a CI job that only audited the marketing homepage while app routes changed in the same release.

Treat them as two separate approvals. Functional QA covers whether things work. Speed readiness covers LCP, INP, and CLS (plus supporting lab metrics such as TBT where INP is noisy) on the URL list you care about after deploy. Both need saved evidence, not a verbal "looks fast."

What pre-deploy performance tests should you run on priority URLs?

Pre-deploy performance testing should cover more than one URL and more than one device type. At minimum, run lab audits on:

Homepage (discovery and first load)
One high-traffic landing or campaign page
One conversion route (pricing, signup, contact, or checkout step)
One content template (blog post, docs page, or help article) if it is built differently

Run each URL on mobile and desktop strategies in PageSpeed Insights or Lighthouse. Record LCP, INP (or TBT on older tooling), CLS, and the Performance score. Compare against your budgets, not only Google's "Good" thresholds if the client SOW is tighter.

Use the environment you will actually ship. A Lighthouse run on localhost misses CDN behaviour, production analytics, and certificate or HSTS differences. A preview URL on the production host (or the live domain before you switch DNS) is the audit you should trust for the publish decision. Staging on a different subdomain is useful for functional QA; it is weaker evidence for speed unless it uses the same tags and caching as production.

Save PDF or JSON exports with timestamps. You will need a baseline to compare against the first production week. Core Web Vitals Monitoring Checklist for Agencies includes onboarding baselines and budget rows you can reuse for a single-site launch.

What does a CI performance check prove before deployment?

If your pipeline already runs Lighthouse CI or similar, treat it as a check on the URLs configured in lighthouserc (or equivalent). A green CI build means: this build did not cross the assertions you defined on those routes in a controlled lab run.

It does not prove:

Every template in the release was in the URL list
Production third-party tags match preview
Field data in Chrome User Experience Report will move within days (CrUX lags)
Mobile traffic mix on the live domain will match the lab profile

CI belongs in the checklist as "pipeline green on configured URLs." It complements pre-deploy manual runs; it does not replace scheduled monitoring after the site is live. How to Set Up Performance Budgets in CI/CD Pipelines walks through assertions and preview hosts if you are adding a check for the first time. A separate note on which checks belong in CI versus ongoing monitoring covers day-to-day ownership for teams that already run pipelines every day.

How do you baseline before ship and compare after deploy?

Before deploy, store a row per URL and strategy: date, environment (preview or production host), LCP, INP, CLS, Performance score, and notes on the LCP element or layout shift source. After deploy, rerun the same URLs at 24 hours and 72 hours on the live domain.

Compare like for like. A mobile lab run before deploy should be compared to mobile lab after deploy, not to desktop field data in Search Console. If scores move, check whether DNS, caching, tag managers, or A/B snippets changed between runs.

If CrUX field data already exists for the domain, record the prior 28-day window but do not expect launch-week fixes to appear immediately. Use lab baselines for the publish decision; use field trends for the first monthly review.

Consent managers, chat widgets, A/B platforms, and analytics often load only on the production domain. A fast staging build can still ship heavy tags on the URL customers hit.

On the production preview (or live domain in quiet hours), open DevTools Performance or Lighthouse and confirm:

Consent banner does not delay LCP element paint beyond budget
Chat or support widgets defer until after first interaction or idle
Tag manager containers match the approved production list, not a staging container
New pixels added for launch week are documented with an owner

If marketing added "just one script" after the last green Lighthouse run, rerun audits before DNS. That single script is a frequent reason checkout passes functional QA and fails INP on mobile.

When is one green Lighthouse run not enough for deployment?

One green Lighthouse score is not enough when any of these are true:

Situation	Why one run fails
Only the homepage was tested	Regressions hide on checkout, account, or search results
Only desktop was tested	Google uses mobile signals for ranking and most retail traffic is mobile
Run happened before final content or tags	Hero media, fonts, or GTM changes after the audit
Scores sit just inside budget	Lighthouse variance can flip a borderline INP or LCP on the next run
CI and manual runs disagree	Investigate URL list, throttling, or auth differences before ship
No baseline file exists	You cannot prove regressions post-launch without a saved before row

Run at least three consecutive lab runs on critical URLs and compare medians when scores are borderline. For high-stakes launches, keep a short rollback plan and the previous build available.

Website ready for deployment checklist (speed)

Use this table beside your functional QA list. Mark each row pass only with evidence (report link, screenshot, or CI job URL).

Check	Pass criteria	Evidence to attach
Priority URLs named	Homepage, conversion route, and one secondary template listed	URL list in ticket or runbook
Mobile lab budgets	LCP, INP, CLS within agreed limits on each priority URL	PageSpeed Insights or Lighthouse export
Desktop lab spot-check	Same URLs on desktop; no desktop-only surprises	Second export per URL
Production or preview host	Audits run on the host that will serve customers, not only localhost	URL bar shows production or approved preview
CI check (if used)	Pipeline green on configured routes for this commit	CI job link
Third-party / consent review	Production tags loaded; no undeclared script after last audit	Tag assistant screenshot or container diff
Baseline stored	Before-deploy metrics saved per URL and device type	Sheet or monitoring tool snapshot
Post-deploy plan	Reruns scheduled at 24h and 72h; alerts configured	Calendar invite or monitor schedule
Rollback named	Person and steps documented if INP or LCP breach after publish	Runbook link

Functional items (forms, SEO meta, redirects, SSL, accessibility spot checks) still belong in your full launch QA. This table is the speed section for "ready for deployment" when people treat green tickets as proof the site is fast.

FAQ

Is staging enough for a speed check before publish?

Staging is enough for functional QA when behaviour matches production. For speed, prefer the production host or a preview URL that uses production CDN, certificates, and tag containers. If staging omits analytics or consent, run a final audit where those scripts load.

Does a 90+ Lighthouse Performance score mean we are ready?

A high score on one URL is encouraging, not sufficient. Confirm priority routes, mobile tests, and third-party scripts on the live host. Scores also weight metrics differently from field Core Web Vitals; treat Lighthouse as a lab check, not a CrUX guarantee.

Should deployment wait for CrUX to turn green?

No for initial launch. CrUX needs traffic and rolls on a 28-day window. Publish when lab baselines and CI checks pass on the production URL, then watch field data and scheduled monitoring in the first weeks.

What to do the week before you switch DNS

Name your priority URLs in writing. Run mobile and desktop lab audits on the production preview and save baselines. Confirm CI assertions cover at least one non-home template if the release touched app routes. Rerun audits after any late tag or hero media change. Schedule post-deploy reruns before you announce the launch.

Run a free domain PageSpeed check on the URLs in your publish list, or start a trial to keep the same routes on a schedule after deploy.

References

Getting started with Lighthouse CI (Chrome Developers)
Web Vitals (web.dev)
Core Web Vitals Monitoring Checklist for Agencies (Apogee Watcher)
How to Set Up Performance Budgets in CI/CD Pipelines (Apogee Watcher)

Originally published on Hashnode.

# The Revolution in Protocol Security: How It Actually Gets Used

Thu, 20 Aug 2026 07:46:14 +0000

A follow-up. The previous article described what we are building. This one describes how it works in practice - and why the shift from academic tool to industrial platform matters.

In a previous article, I described the architecture of a protocol analyzer we are building: automatic repair, sixteen attacker models, GNN-guided search, modular crate design. That was the "what." This article is the "how" - how this tool gets used, what it changes in practice, and why we believe it represents a shift from laboratory instrument to industrial platform.

The problem with existing tools is not only capability - it is also access

ProVerif and Tamarin are remarkable achievements in formal methods. They are also laboratory instruments by design. You install them on a research machine, write a model in their DSL, run analysis in a terminal, read the output. This workflow is fine for a PhD student. It does not work for a security engineer at a company that ships TLS implementations.

The gap concerns where these tools live in the development lifecycle, not what they can prove. Existing tools sit outside CI pipelines, outside IDEs, outside deployment workflows. They are consulted - if at all - after a protocol is designed, before it ships. Rarely during development, rarely after deployment.

We are building something that lives inside the lifecycle.

CI/CD integration: verify on every commit

The analyzer ships as a GitHub Action. Every time a protocol description changes - a new message flow, a modified key exchange, an added authentication step - the analyzer runs automatically. It checks whether the change introduces a vulnerability. If it does, the action fails with a concrete attack trace and a proposed fix.

This is not a theoretical capability. The action is designed to run in minutes on standard CI hardware, not hours on a research cluster. The performance targets we described in the previous article - 10x to 50x speedup, sub-ten-minute analysis for real-world protocols - exist to make this practical. A tool that takes six hours to analyze a protocol cannot run in CI. One that takes six minutes can.

Beyond GitHub Actions, the analyzer provides a CLI that integrates with any CI system - GitLab CI, Jenkins, Buildkite. The output is machine-readable: SARIF for security scanners, JSON for custom pipelines, JUnit XML for test reporting. Protocol verification becomes another test suite in the pipeline.

IDE integration: find vulnerabilities while you write

The analyzer ships a Language Server Protocol (LSP) implementation. In a VS Code editor, this means: syntax highlighting for the protocol DSL, inline error reporting as you type, hover documentation for security properties, and one-click analysis from the editor.

The experience is analogous to writing Rust with rust-analyzer: the tool understands what you are writing and gives feedback in real time. When you define a key exchange step, the analyzer checks whether the key material is properly authenticated. When you add a new message, it checks whether the freshness guarantees hold. The vulnerability is flagged before you commit the code - not after.

Real-time analysis: the daemon and the sniffer

This is where the tool stops being a verifier and becomes a platform.

The analyzer runs as a background daemon with a gRPC API. Any service in the infrastructure can send a protocol description and receive analysis results. But the daemon does more than wait for requests - it listens to live traffic.

The sniffer module captures network traffic (PCAP), reconstructs protocol sessions from packet streams, and feeds them to the symbolic engine in real time. When a TLS handshake happens on the network, the analyzer checks whether that handshake - with its specific cipher suite, extensions, and parameters - satisfies the security properties. When an unusual pattern appears - a repeated nonce, an unexpected message order, a downgrade attempt - it flags the anomaly.

This is live verification - analysis that runs while the protocol is executing, not after the fact.

The real-time constraint changes the engineering at its core. Traditional protocol analysis explores the full state space - every possible interleaving, every attacker action. That takes hours. Real-time analysis uses a bounded model: explore the state space up to depth N, report what you find, continue monitoring. The GNN-guided search from our previous article is critical here - it prioritizes the most promising exploration paths, finding attacks faster in bounded time.

Web interface: collaboration, not just output

The analyzer ships a web UI. This is a requirement for adoption, not a luxury.

Security review is a collaborative process. An analyst finds an attack, a protocol designer evaluates the proposed fix, a manager decides whether the risk is acceptable. In existing tools, this workflow happens outside the tool: screenshots in Slack, traces in emails, decisions in meetings.

The web UI makes the analysis itself collaborative. Attack graphs are rendered in 3D - navigable, zoomable, inspectable. Repair proposals are presented side-by-side with the original protocol, with the differences highlighted. Reviewers can comment, approve, or reject fixes. The entire security review happens inside the platform.

Converters: meeting people where they are

The analyzer accepts input from ProVerif, Tamarin, Scyther, Avispa, CryptoVerif, and Maude descriptions. This is an adoption strategy, not a convenience feature.

Thousands of protocols are already modeled in these languages. Researchers and engineers who have invested years in ProVerif models should not have to rewrite them. They import their existing description, run the analyzer, and get the same output plus repair proposals plus real-time monitoring plus CI integration.

The converters are imperfect - each source language has constructs that do not map cleanly. But they handle the common case well enough to lower the barrier from "I should try this tool" to "I just tried this tool."

Platform, not tool

Let us be explicit about what we are building. A tool does one thing from the command line. A platform integrates into workflows, provides APIs, runs continuously, and collaborates with other systems.

The protocol analyzer we are building has:

A CLI for direct use and scripting
A GitHub Action for CI/CD
An LSP server for IDE integration
A gRPC API for programmatic access
A daemon for continuous monitoring
A sniffer for real-time traffic analysis
A web UI for collaborative review
WASM bindings for browser-based playground
Converters from five existing tools

This is a platform. The symbolic analysis core - the part described in the previous article - is the engine. Everything else is the interface between that engine and the world where protocols are built, deployed, and attacked.

The revolution is practical, not technical

The technical contributions matter: automatic repair, GNN-guided search, incremental closure, sixteen attacker models. But the reason they matter is that they enable a new capability - protocol verification that runs in CI, in your IDE, on your network, in real time.

ProVerif and Tamarin proved that symbolic protocol analysis is possible. We are trying to prove that it is practical - practical enough to run on every commit, practical enough to catch vulnerabilities while you type, practical enough to monitor live traffic.

That is the shift: from laboratory instrument to industrial platform. From "analyze after design" to "verify during development and after deployment." From a tool that finds attacks to a platform that finds them, fixes them, and watches for them.

Status

The CI/CD action, the LSP server, the daemon, and the sniffer are in active development. The web UI has working prototypes. The converters handle common cases. The core engine - described in the previous article - is functional and stable.

This project is under active development. The platform described here is being built, not shipped. But the direction is clear, and the architecture supports it.

This project is under active development. This article describes the planned platform capabilities of a tool still being built. The previous article covers the core engine architecture; this one covers the interfaces between that engine and the development lifecycle.

Follow the project: github.com/Sup4ikX - a preview release will be published there as soon as the first production version is ready.

Blue/green deployments on Kubernetes with Argo Rollouts

Tue, 21 Jul 2026 13:03:23 +0000

One of the harder questions to answer at scale is how to ship without a few seconds where your users are getting timeouts or your fleet is split across two image versions.

Kubernetes’s default rolling update strategy gradually deploys new pods and retires old ones, but during the swap, your service runs both versions side by side, and a regression in the new image affects every request that lands on a new pod.

Progressive delivery patterns like blue/green have long existed: you stand the new version up alongside the old, prove it’s healthy on a separate preview endpoint, then flip user traffic across. Blast radius shrinks to nothing in the bad case, rollback is a single command, and your release stops being a held-breath moment.

In this post, you’ll set that up with Argo Rollouts, the controller behind progressive delivery in the Argo ecosystem (which graduated from the CNCF in 2022).

What is Argo Rollouts (and why you need it)

Argo Rollouts is a Kubernetes controller and a set of CRDs that bolt blue/green, canary, and other progressive delivery strategies onto your cluster. The primary CRD is Rollout, a drop-in replacement for the standard Deployment.

You convert an existing Deployment by changing the apiVersion to argoproj.io/v1alpha1 and the kind to Rollout, then adding a strategy.blueGreen or strategy.canary block that describes how a new revision should roll out.

A major reason to reach for it is that there’s no built-in way to do this kind of traffic control in Kubernetes (you can’t decide where requests go independently of which pods are Ready). There’s no easy rollback to the previous version once the update has started.

Argo Rollouts fills in everything around that. It plugs into ingress controllers (Traefik, ALB) and service meshes (Istio, Linkerd, SMI) for real traffic shaping. It can query metrics providers (Prometheus, Datadog, CloudWatch, New Relic) to gate promotions on hard numbers, and it tracks every revision as its own ReplicaSet, so flipping back is instant.

Prerequisites

This tutorial assumes some familiarity with Kubernetes. You’ll also need:

A working Kubernetes cluster (EKS, GKE, AKS, or local like Minikube/Kind).
kubectl installed and pointed at the cluster (kubectl get nodes should return at least one Ready node)
helm v3 installed
curl for hitting the demo app

Step 1: Installing the Argo Rollouts controller

Argo Rollouts ships as a controller that runs in its own namespace, along with a kubectl plugin you’ll use on your laptop to inspect and steer rollouts.

To install the controller, follow these steps:

Add the Argo Helm repo and install the controller:

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argo-rollouts argo/argo-rollouts \
  --namespace argo-rollouts \
  --create-namespace \
  --wait

Confirm the controller pods are up:

kubectl -n argo-rollouts get pods

You should see something like:

NAME                             READY   STATUS    RESTARTS   AGE
argo-rollouts-dcd465dfc-8m2ql    1/1     Running   0          79s
argo-rollouts-dcd465dfc-q92k4    1/1     Running   0          79s

Step 2: Installing the kubectl argo rollouts plugin

The controller is running, but the most ergonomic way to drive a rollout — inspecting state, setting images, promoting, rolling back — is the kubectl argo rollouts plugin. It’s a separate binary that drops onto your PATH, and kubectl picks it up automatically.

To install the plugin, follow these steps:

On macOS with Homebrew:

brew install argoproj/tap/kubectl-argo-rollouts

On Linux:

curl -sLO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x kubectl-argo-rollouts-linux-amd64
sudo mv kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts

Verify it’s wired up:

kubectl argo rollouts version

You should see something like kubectl-argo-rollouts: v1.8.3+.... From here on, we’ll use kubectl argo rollouts … subcommands to drive the rollout.

Step 3: Defining the Rollout

Argo Rollouts’ core idea is that you swap your Deployment for a Rollout resource. The pod template inside it is identical to a Deployment’s.

What changes is the spec.strategy block, which describes how a new revision should roll out.

For blue/green, you need three things:

A Rollout with spec.strategy.blueGreen configured
An active Service, which always points to whichever ReplicaSet is currently serving production traffic
A preview Service, which points at the new ReplicaSet before it gets promoted, so that you can test it in isolation

Argo Rollouts injects the rollouts-pod-template-hash label into each Service’s selector at runtime, which is how it switches traffic without you ever editing the Services.

Write the manifest:

cat > rollout.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo-active
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  selector:
    app: rollouts-demo
---
apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo-preview
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  selector:
    app: rollouts-demo
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollouts-demo
spec:
  replicas: 2
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: rollouts-demo
  template:
    metadata:
      labels:
        app: rollouts-demo
    spec:
      containers:
        - name: rollouts-demo
          image: argoproj/rollouts-demo:blue
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          resources:
            requests:
              cpu: 25m
              memory: 32Mi
  strategy:
    blueGreen:
      activeService: rollouts-demo-active
      previewService: rollouts-demo-preview
      autoPromotionEnabled: false
      scaleDownDelaySeconds: 30
EOF

A few fields are worth calling out in the blueGreen block:

activeService and previewService are the names of the two ClusterIP Services above. Argo Rollouts owns their selectors from here on; you don’t edit them by hand.
autoPromotionEnabled: false is what makes this a manual promotion. The new ReplicaSet comes up, you inspect it on the preview Service, and only then do you flip the active Service over. Set it to true (the default) and Argo will auto-promote the moment the new pods are Ready.
scaleDownDelaySeconds: 30 keeps the old (blue) ReplicaSet around for 30 seconds after promotion, so if something goes wrong in those first few seconds, you can flip back instantly without rescheduling pods.

Apply it:

kubectl apply -f rollout.yaml

We’re using argoproj/rollouts-demo, a tiny app published by the Argo team that serves an HTML dashboard and a /color endpoint that reports which tagged image is running (blue, green, yellow, etc.). It’s perfect for seeing the cutover happen in real time.

Step 4: Checking the initial state

Take a look at the rollout:

kubectl argo rollouts get rollout rollouts-demo

Output:

Name:            rollouts-demo
Namespace:       default
Status:          ✔ Healthy
Strategy:        BlueGreen
Images:          argoproj/rollouts-demo:blue (stable, active)
Replicas:
  Desired:       2
  Current:       2
  Updated:       2
  Ready:         2
  Available:     2

NAME                                       KIND        STATUS     AGE  INFO
⟳ rollouts-demo                            Rollout     ✔ Healthy  34s
└──# revision:1
   └──⧉ rollouts-demo-86c957c6d6           ReplicaSet  ✔ Healthy  34s  stable,active
      ├──□ rollouts-demo-86c957c6d6-72kjf  Pod         ✔ Running  34s  ready:1/1
      └──□ rollouts-demo-86c957c6d6-nv3zg  Pod         ✔ Running  34s  ready:1/1

One revision, two pods, both stable and active. Both Services currently point at the same ReplicaSet hash. You can confirm with:

kubectl get svc rollouts-demo-active rollouts-demo-preview \
  -o jsonpath='{range .items[*]}{.metadata.name}{" -> hash="}{.spec.selector.rollouts-pod-template-hash}{"\n"}{end}'

Output:

rollouts-demo-active -> hash=86c957c6d6
rollouts-demo-preview -> hash=86c957c6d6

Step 5: Triggering a new version

Now let’s deploy a new revision. We’ll change the image tag from blue to yellow:

kubectl argo rollouts set image rollouts-demo \
  rollouts-demo=argoproj/rollouts-demo:yellow

Argo creates a new ReplicaSet (rev 2) for the yellow image and waits, because we set autoPromotionEnabled: false. The active Service still points at blue. The preview Service is re-pointed at yellow:

kubectl argo rollouts get rollout rollouts-demo

Output:

Status:          ॥ Paused
Message:         BlueGreenPause
Strategy:        BlueGreen
Images:          argoproj/rollouts-demo:blue (stable, active)
                 argoproj/rollouts-demo:yellow (preview)
Replicas:
  Desired:       2
  Current:       4
  Updated:       2
  Ready:         2
  Available:     2

NAME                                       KIND        STATUS     AGE  INFO
⟳ rollouts-demo                            Rollout     ॥ Paused
├──# revision:2
│  └──⧉ rollouts-demo-7cf9dff6bb           ReplicaSet  ✔ Healthy  38s  preview
│     ├──□ rollouts-demo-7cf9dff6bb-cbp2c  Pod         ✔ Running  38s  ready:1/1
│     └──□ rollouts-demo-7cf9dff6bb-fn2gt  Pod         ✔ Running  38s  ready:1/1
└──# revision:1
   └──⧉ rollouts-demo-86c957c6d6           ReplicaSet  ✔ Healthy  5m   stable,active
      ├──□ rollouts-demo-86c957c6d6-72kjf  Pod         ✔ Running  5m   ready:1/1
      └──□ rollouts-demo-86c957c6d6-nv3zg  Pod         ✔ Running  5m   ready:1/1

This is the heart of blue/green. The cluster is now running both versions, but only blue is serving real traffic.

Step 6: Proving the split with curl

Forward both Services to your laptop on different local ports:

kubectl port-forward svc/rollouts-demo-active 8080:80 >/dev/null 2>&1 &
kubectl port-forward svc/rollouts-demo-preview 8081:80 >/dev/null 2>&1 &
sleep 3

Hit each one using:

echo "active : $(curl -s http://127.0.0.1:8080/color)"
echo "preview: $(curl -s http://127.0.0.1:8081/color)"

Output:

active : "blue"
preview: "yellow"

This is exactly the window where you’d run smoke tests, point a staging frontend at the preview hostname, or have Argo run an AnalysisTemplate against Prometheus.

Nothing about production traffic has changed yet.

When you’re done with the port-forwards, run the following command:

kill %1 %2 2>/dev/null

Step 7: Promoting

When you’re happy, flip the active Service over with one command:

kubectl argo rollouts promote rollouts-demo

Output:

rollout 'rollouts-demo' promoted

Argo updates the active Service’s selector to the new ReplicaSet hash.

Subsequent requests should show production traffic is in yellow. The old blue pods stick around for scaleDownDelaySeconds (30 by default) before being torn down, which is what makes the next section possible.

Confirm the cutover by running:

kubectl argo rollouts status rollouts-demo --timeout 60s

You should see Healthy, and the Service selectors should now agree:

kubectl get svc rollouts-demo-active rollouts-demo-preview \
  -o jsonpath='{range .items[*]}{.metadata.name}{" -> hash="}{.spec.selector.rollouts-pod-template-hash}{"\n"}{end}'

Output:

rollouts-demo-active -> hash=7cf9dff6bb
rollouts-demo-preview -> hash=7cf9dff6bb

Step 8: Rolling back

If something goes wrong after the promotion (a metric tanks, you spot an error in the logs, a teammate flags a bug), undo it by running:

kubectl argo rollouts undo rollouts-demo

That brings the previous ReplicaSet back as the new “preview” and pauses, waiting for you to confirm with promote once more, which flips the active Service back to it. Because the old pods were kept warm by scaleDownDelaySeconds, this happens in seconds, not whatever your image pull time is.

Step 9: Cleaning up

Once you’re done, you can tear down the demo using:

kubectl delete -f rollout.yaml
helm uninstall argo-rollouts -n argo-rollouts
kubectl delete ns argo-rollouts

How does this fit with Octopus Deploy and Argo CD

Everything we’ve done so far works on its own. You’ve got a Rollout, two Services, and a one-command promote/undo loop.

Argo Rollouts is happy to do its job at the cluster level. What it doesn’t have is an opinion on how dev becomes staging and then becomes production. Who’s allowed to push the button, or what the deployment history looked like six weeks ago.

This is the layer Octopus Deploy is built for. A good mental model is:

Argo Rollouts owns the cluster-side mechanics: Which ReplicaSet is active, which is preview, when to flip, and when to scale down old pods.
Argo CD owns the GitOps sync: The Rollout (and its Services) live in a Git repo, and the cluster state is reconciled to match.
Octopus owns everything above that: Environments, approval gates, release lifecycles, audit trails, and the self-service UI that developers actually click on.

The promotion path between environments is described once in Octopus and reused across every service, instead of being re-encoded in each team’s CI script.

Ship green, sleep through the night

If you made it this far, you’ve got the cluster-side mechanics of progressive delivery sorted: a Rollout flipping between active and preview Services, a manual promotion gate, and instant rollback. That’s the hard, hands-on layer done.

What’s missing is the orchestration above it, including environments, approvals, audit trails, and the self-service flow your developers actually click. That’s where Octopus Deploy slots in, sitting on top of Argo CD and Argo Rollouts to give you a complete progressive delivery stack across every environment, not just one cluster.

Connect your Argo CD instance to Octopus and see how the whole pipeline comes together, or try Octopus free and wire it up against your own cluster.

Happy deployments!

When Lighthouse CI maintenance in CI/CD pipelines becomes a second job

Sat, 04 Jul 2026 08:27:36 +0000

Performance budget gates in GitHub Actions are free until someone owns twelve client workflows, Chrome drift, and client reporting. How agencies split Lighthouse CI from managed PageSpeed monitoring.

The Slack thread started with a screenshot of a green GitHub Actions run. By the third reply someone had pasted a Lighthouse JSON artefact, a link to a Chrome release note, and a question nobody wanted to own: "Which client repository still pins Lighthouse 10?"

That is the week Lighthouse CI stopped being a merge gate and became a second job. The pipeline still passed and the portfolio still needed evidence, but the difference was who paid in hours: the developer shipping a feature, or the one person who inherited every lighthouserc file across client repositories.

When does Lighthouse CI maintenance outgrow a CI/CD pipeline?

Lighthouse CI earns its place early. You wire assertions on a preview URL, block a CLS regression, and the team trusts the red build. The cost is front-loaded configuration, not ongoing calendar time.

The shift happens when success creates obligations a CI/CD pipeline was never designed to carry:

Every new client repository needs a copied workflow, pinned Chrome, and preview URL rules that match their host.
Assertions need tuning after flaky LCP on cold runners, so thresholds loosen until they barely catch real regressions.
Account managers ask for client-ready reports, and the only export is a CI/CD artefact someone must turn into slides.
Production URLs outside the two preview paths regress while the job stays green.

At that point you are not "running Lighthouse CI in a pipeline." You are operating a small internal product: version pins, runner hygiene, assertion policy, and reporting glue. For a single product team that can be fine. For an agency portfolio it competes with billable delivery.

How do you know Lighthouse CI in CI/CD became an unpaid side role?

We treat these as signals to shrink CI/CD scope or add a managed monitoring layer, not as moral failure. Teams hit them around five to fifteen client sites, sometimes sooner when preview hosts differ wildly.

Flaky Lighthouse CI runs on GitHub Actions

Engineers merge after the third "Re-run failed jobs" click because INP moved six points on a shared runner. When the gate is noise, people route around it.

@lhci/cli and Lighthouse version drift across repositories

One repository uses @lhci/cli 0.13, another 0.14, Lighthouse 11 on one client and 10 on another. Nobody can say which assertion set is authoritative without opening six pull requests.

Preview URL versus production: green CI/CD, red PageSpeed Insights

The CI/CD job guards / on a Netlify preview while production serves /checkout from a different CDN rule. Sponsors see red on PageSpeed Insights; developers point at a green pipeline run and both sides are partially right.

Lighthouse CI artefacts instead of client-ready performance reports

Monthly retainers need a short narrative: which URLs broke budget, when, and what changed. If the answer lives in downloaded JSON and a manual chart, someone is doing monitoring work outside the pipeline, unpaid.

One engineer as the default Lighthouse CI maintainer

One senior developer becomes "the Lighthouse person," fixes workflows between client calls, and resents every Chrome update. That is a job description, not a YAML tweak.

If three or more patterns sound familiar, Lighthouse CI is already a second job. The fix is not deleting the pipeline check. It is deciding which hours belong in merge protection and which belong in scheduled monitoring with client-ready output.

Which performance budget checks should Lighthouse CI enforce in CI/CD?

We keep the CI/CD slice thin on purpose. Merge protection should answer a narrow question: did this candidate build break budgets on the one or two preview URLs we trust for this repository?

Keep as errors when the check is stable and the developer on the pull request can fix the cause:

CLS on in-scope templates where layout is the likely regression.
Bundle or script byte ceilings when the build exposes an artefact.
Total blocking time or main-thread proxies when the change touches JavaScript entry points.

Warn, do not block, when preview parity is weak: cold CDN LCP, ads on staging, INP without enough repetitions to trust a median.

Do not ask CI/CD to replace production monitoring: field CrUX exports, full sitemap crawls, rank proxies, or every revenue template on every docs-only commit. Those checks belong on a schedule against production URLs, with alerts that account managers can forward.

The wiring patterns, assertion keys, and GitHub Actions sketch live in our step-by-step guide: How to set up performance budgets in CI/CD pipelines. Use that guide to implement a small gate; use the criteria above to decide how small.

Lighthouse CI versus managed monitoring: why agencies hit build-versus-buy pressure early

Lighthouse CI is open source and runner-native, which makes it the default first step. Managed PageSpeed monitoring adds invoice line items, which makes teams delay the decision until the hidden invoice arrives: senior hours spent keeping workflows honest.

The build path scales poorly across repositories because CI/CD is repository-centric and agencies are portfolio-centric. Copying configuration per client multiplies drift. Preview URL changes when a client moves hosts. Authentication on staging breaks collection until someone scripts login. Each exception lands on the same maintainer.

The buy path trades some control for repetition: scheduled lab runs across many URLs, history, thresholds shared with client reports, and alerts when production templates cross budget. It does not replace merge gates on preview; it absorbs the work a pipeline should never have carried: full URL lists, post-deploy cadence, and evidence sponsors can read without terminal access.

We wrote the comparison for agency leads evaluating that trade explicitly: Lighthouse CI versus managed monitoring: build versus buy for agencies. The headline question is not "free versus paid." It is whether your team wants to operate monitoring infrastructure or operate client sites.

How to shrink Lighthouse CI scope in CI/CD pipelines without losing merge protection

When CI/CD became a second job for us, the recovery plan had four moves. None required removing Lighthouse from the pipeline entirely.

Limit Lighthouse CI to two preview URLs per client repository

Two preview paths maximum on small retainers: homepage plus one revenue or app-shell template agreed in the statement of work. Everything else lives in scheduled monitoring.

One performance budget document for CI/CD gates and client reports

Copy numbers from the same budget document monitoring uses. If CI/CD gates and the client report disagree on thresholds, engineers ignore both.

Demote flaky LCP to warnings; keep CLS and bundle size as errors

Demote flaky LCP to warnings until preview matches production behaviour on paper. Keep CLS and bundle ceilings as errors people act on.

Document pre-merge CI/CD versus post-deploy monitoring in the README

CI/CD answers pre-merge; monitoring answers post-deploy. Write that split in the repository README so account teams stop asking the pipeline to prove checkout on Friday's release.

Layer managed monitoring beside the CI/CD job rather than inflating YAML until every pull request waits ten minutes on Chrome. Multi-tenant monitors exist precisely because portfolio evidence does not belong inside each client's GitHub Actions bill.

Lighthouse CI in CI/CD versus scheduled PageSpeed monitoring: agency split for one repository

For a typical agency client on a modern JavaScript stack, we aim for under ten minutes of CI/CD web performance time and zero manual JSON exports per month.

Phase	Tooling	Question
Pre-merge	Lighthouse CI plus one bundle guard	Did this branch regress CLS or script weight on the two preview URLs in scope?
Post-deploy	Scheduled PageSpeed monitoring	Are production templates still inside budget on the cadence we promised?
Reporting	Monitoring product export	What do we send the sponsor without opening a terminal?

Green in CI/CD and red in the Friday report means the split failed, not that someone lied. Either preview parity is wrong, the URL list in monitoring is right and the pipeline was too narrow, or the regression shipped through a path CI/CD never watched. All three are operational fixes, not reasons to add a fourth Lighthouse run on every push.

Audit Lighthouse CI maintenance before you copy the workflow to another client

Pick the client repository where the CI/CD job failed most often last quarter. Answer honestly:

Who last updated lighthouserc and the Chrome pin?
How many hours went to flaky re-runs versus feature work?
Which production URL regressed while the pipeline stayed green?
What did the sponsor receive that was not a raw artefact?

If the maintainer has a name and the sponsor answer is "a screenshot from PageSpeed Insights," you already have a second job. Implement the thin CI/CD template from performance budgets in CI/CD pipelines, then read Lighthouse CI versus managed monitoring before copying the workflow into client seven.

Merge gates should block merges you would roll back. Everything else belongs on a schedule, with owners, on the full URL list the retainer names.

Originally published on Hashnode.

Ninja is Slow. We Built a Build System in Go That Outperforms It by 39x on a Quarter-Million Tasks

Fri, 03 Jul 2026 13:10:46 +0000

ForgeZero 2.0: We Built a Build System 39x Faster Than Ninja

And yes, it's open source. No bullshit, just architecture.

The Problem Everyone Ignores

You've been there. Sitting at your desk, coffee getting cold, waiting for your build to finish. 10 seconds. 30 seconds. A minute.

CMake parses its DSL. Ninja forks 2,000 processes. Make thrashes the disk. GNU Autotools exists, and everyone pretends not to notice.

This isn't a feature gap. This is architectural malpractice.

Every major build system was designed in an era when parallelism meant "spawn processes." That's it. No thought given to the actual cost—fork syscalls, context switching, memory duplication, I/O bottlenecks.

We decided to fix it properly.

What ForgeZero Actually Is

ForgeZero is a cross-platform build system for C, C++, Objective-C, and Assembly. Written in Go. Single binary. No configuration files needed.

Point it at a directory:

fz build -o binary

It detects your source files, selects compilers (GCC, Clang, Zig), picks assemblers (NASM, GAS, FASM, or embedded), and outputs production-ready binaries, libraries, or ISO images.

That's literally it.

No Makefile. No CMakeLists.txt. No autotools scripts. One command. Full parallelism. Reproducible builds by default.

Repository: https://github.com/forgezero-cli/forgezero
License: GPLv3 (fully open source)

The Benchmark That Broke Ninja's Narrative

Real project. 2,000 object files. 1,000 .c source files. Real hardware.

ForgeZero:  258 ms  (User: 348ms, System: 520ms)
Ninja:      10.1 s  (User: 73.9s, System: 47.8s)

Speedup: 39.2x ✓

This isn't synthetic. This isn't a contrived test case. This is what happens when you link 2,000 real object files on a real codebase.

The gap grows with scale:

File Count	ForgeZero	Ninja	Speedup
1,000	~85ms	~2.7s	32x
2,000	~258ms	~10.1s	39x
10,000	~1.2s	~50s	~42x

Ninja's overhead scales linearly with file count. ForgeZero's scales logarithmically. The larger your codebase, the more you win.

Watch the actual benchmark: https://t.me/wienton/176

Why This Demolishes Every Competitor

1. Architecture: One Process, Infinite Goroutines

Traditional systems:

foreach file:
  fork() → new process
  compile
  wait for completion
  write .o to disk
  read .o back for linking

ForgeZero:

spawn N goroutines (microseconds, not milliseconds)
all 2000 files compile in parallel
all objects stay in memory
linker reads from memory arena
zero context switching overhead

The difference: Goroutine creation = microseconds. Process fork = milliseconds. At 2,000 files, you're talking about 1-2 seconds saved just on process overhead.

2. Zero Allocations in Hot Paths

This is the part other build systems don't even think about.

BenchmarkLinkerHotPath:  0 allocs/op, 0 B/op
BenchmarkCodegenHotPath: 0 allocs/op, 0 B/op
BenchmarkHashBLAKE3:     0 allocs/op, 0 B/op

No garbage collection pauses. No "stop-the-world" events. Deterministic latency like C or Rust.

Achieved through:

Pre-allocated arenas for intermediate objects
Stack buffers for I/O operations
sync.Pool for temporary structures
Zero make(), append(), or new() in critical sections

When you're linking thousands of files, GC pauses compound. We eliminated them entirely.

3. Memory-Resident Object Cache

Ninja writes every .o file to disk, then reads it back. Disk I/O at SSD speeds is 500 MB/sec.

ForgeZero keeps objects in memory: 10+ GB/sec throughput.

On a 2000-file project, this alone saves 200-300ms.

4. BLAKE3 Instead of SHA-256

Caching depends on file hashes. SHA-256 is cryptographically solid but slow.

BLAKE3 is:

7x faster than SHA-256
Parallel (SIMD accelerated)
Cryptographically modern

On 2000 files: SHA-256 (~500ms) vs BLAKE3 (~70ms). That's 430ms saved per rebuild.

5. Custom Lock-Free Scheduler

ForgeZero doesn't rely on the OS scheduler. It implements its own:

Mutex, Semaphore, WaitGroup (no external dependencies)
Lock-free ring buffer for task queues (CAS atomics)
Worker pool with goroutine reuse
Atomic counters (no mutex contention)

Even under load, zero contention. Maximum throughput.

What's New in ForgeZero 2.0 (v5.3.0)

Embedded x86 Assembler

NASM is gone. ForgeZero now includes a native Intel-syntax assembler that generates correct ELF64 object files.

One dependency eliminated
Works on all platforms (Linux, Windows, macOS)
Same behavior everywhere

ISO Image Generation

fz build -iso -iso-out kernel.iso
qemu-system-x86_64 -cdrom kernel.iso

For kernel and firmware developers: no more mkisofs scripts. One flag. Bootable ISO. Done.

Support for -iso-hybrid (BIOS + UEFI).

Three Cache Modes

fz build                  # disk mode (default, persistent)
fz build -cache=ram       # memory mode (CI/CD, ephemeral envs)
fz build -cache=off       # no cache

Perfect for Docker containers where disk is slow or ephemeral.

Inline Bash Scripts

scripts:
  - bash: echo "Building..."
  - bash: ./configure --prefix=/usr
  - bash: make install

If no shell exists, uses embedded minimal runner (cd, export, direct binary calls).

Rollback Support

fz --rollback              # Revert to previous version
fz --rollback-to v5.2.0    # Revert to specific version

Production-grade versioning. If something breaks, you recover in seconds.

Rewritten Scheduler

Removed external workerpool dependency. Built custom with:

Better error aggregation
Context-based cancellation
Graceful shutdown
Full race condition safety

Rewritten Config System

Low-allocation merging (no unnecessary copies)
Parallel config loading
Clear merge semantics (strings override, slices deduplicate, maps merge)
Safe defaults with validation

Cross-Platform Reality

Single codebase. Three platforms. Identical behavior.

Linux:

Direct syscalls via golang.org/x/sys/unix
SYS_OPENAT, SYS_UNLINKAT
No libc

Windows:

Native implementation via golang.org/x/sys/windows
CreateFile, ReadFile, WriteFile APIs
Retry logic for file locks

macOS:

Full Darwin syscall support
Identical behavior to Linux

VFS Layer abstracts all OS differences:

Path/separator normalization
TOCTOU attack prevention
Symlink boundary enforcement (security by default)

Built-In Security (Not Bolted On)

fz audit — SAST Scanner

fz audit              # Scan for all issues
fz audit --strict     # Strict mode

Detects:

Secrets (AWS keys, GitHub tokens, API keys)
Dangerous patterns (gets, sprintf without bounds, alloca in loops)
License violations (GPL/AGPL in proprietary projects)
Known CVEs in dependencies

fz sbom — Software Bill of Materials

fz sbom --output sbom.xml

CycloneDX format with BLAKE3 hashes for every component. Essential for aerospace, medical, and fintech systems.

fz verify — Source Integrity

fz verify --manifest .fz.manifest

Checks every file against BLAKE3 manifest. Detects corruption and supply chain attacks.

--reproducible — Bit-for-Bit Reproducibility

fz build --reproducible

Two builds of identical code produce identical binaries (down to the last byte):

Build ID suppressed
Timestamps zeroed
Paths normalized
Order deterministic

Verifiable that no one modified your source.

fz doctor — Environment Diagnostics

fz doctor

Checks:

Compiler availability and versions
File access permissions
Platform integrity
Required dependencies

Human-readable report of what works and what doesn't.

Supported Everything

Category	Details
Languages	C, C++, Objective-C, Assembly (NASM/GAS/FASM), Gloria (custom bare-metal language)
Output	ELF, PE, Mach-O, static libraries, shared libraries, flat binaries, ISO images
Cross-Compilation	ARM (32/64), RISC-V, WebAssembly, Windows, full support via Zig toolchain
Tools	`-compile-commands` (LSP), `-watch` (file monitoring), `-shell` (REPL), `-bench` (profiling), `-debug` (DWARF symbols)
Advanced	Bare-metal linking, custom linker scripts, freestanding environments, BIOS+UEFI support

Package Manager (fz pm)

fz pm add <repo>[@version]    # Add from Git
fz pm remove <package>         # Remove
fz pm list                     # List installed
fz pm catalog                  # Browse official catalog
fz pm search <query>           # Search
fz pm install <name>           # Install with BLAKE3 verification

All packages verified with BLAKE3 before installation. No npm dependency hell.

The Numbers Don't Lie

976 commits in 2 months (16 commits/day—this is sprint-grade work)
42,048 lines of code (32,275 Go, rest tests + docs)
234 Go files
90%+ test coverage on critical packages
0 allocations in hot paths (benchmarked)
100% golangci-lint in strict mode
Passes all -race checks (no data races)
Works on Linux, Windows, macOS with zero code changes

This is not fantasy. This is production-grade code.

Comparison: The Reality Check

Feature	Make	Ninja	CMake	ForgeZero
Config files	Yes	Yes	Yes	No
Caching	No	Limited	No	3 modes
Parallelism	Processes	Processes	Processes	Goroutines
Zero-alloc paths	No	No	No	Yes
Embedded assembler	No	No	No	Yes
ISO images	No	No	No	Yes
Package manager	No	No	No	Yes
SAST/SBOM	No	No	No	Yes
Cross-compilation	Limited	Limited	Limited	Full
Speed	Baseline	Fast	Medium	39x Ninja
Windows native	Limited	Limited	Yes	Yes
Rollback support	No	No	No	Yes
Reproducible builds	No	No	No	Yes

One system dominates every dimension.

System Requirements

OS: Linux, Windows (native or WSL2), macOS
Architecture: amd64, arm64
Optional: GCC/Clang (C/C++), Zig (cross-toolchain), Git (package manager)
For ISO: xorriso (separate installation)

Everything else is embedded in a single binary.

Why This Matters: The Reality

Speed Changes Workflow

Before: 10 seconds per rebuild = context lost
After:  250 milliseconds per rebuild = never interrupt flow

This isn't productivity theater. This is fundamental work process improvement.

On a 50-engineer team, if each engineer rebuilds 50 times per day:

Ninja:      10s × 50 builds × 50 engineers = 41,666 seconds wasted
ForgeZero:  0.25s × 50 builds × 50 engineers = 1,041 seconds wasted

Saved: 40,625 seconds/day = 11.3 person-hours/day

That's literally one full-time engineer's output, reclaimed.

No Config = No Errors

CMakeLists.txt, Makefile, autotools—these are programming languages with their own syntax, semantics, and pitfalls.

ForgeZero needs zero configuration to work. YAML if you want customization. That's it.

New developers don't spend a day figuring out the build system. They fz build. Done.

Systems Work Is Irreplaceable

This tool exists because systems programming is fundamentally different. You can't CMake your way out of bare-metal development.

Kernel developers need embedded assembly → we built it in
Firmware developers need ISO images → integrated
Security-critical projects need SBOM and reproducible builds → bundled
Embedded devs need cross-compilation → fully supported

We're not trying to replace Gradle or Maven. We're building the right tool for systems work.

The Roadmap

We're not done:

Linux Kbuild support for kernel compilation
Enhanced IDE integration (VSCode, CLion)
Gloria language expansion (custom systems language)
Further performance optimization (targeting 50x+ speedup)
Additional architecture support (PowerPC, MIPS)

This is a living project. 976 commits in 2 months. Not a one-off. Not a side project. Real engineering work.

Who This Is For

You're a Systems Programmer

Building kernels, bootloaders, or firmware
Need deterministic, reproducible builds
Want security integrated, not bolted on
Tired of Makefile spaghetti

This is for you.

You Manage a Build Infrastructure

Your CI/CD pipeline is 60% waiting for builds
You want to scale without renting more cloud resources
You need reproducible, verifiable builds

This saves money and time.

You Care About Simplicity

You believe configuration is a source of bugs
You want tools that work zero-config and scale up
You respect your developers' time

This respects your choices.

Get Started

# Linux/macOS
curl -sSL https://raw.githubusercontent.com/forgezero-cli/ForgeZero/main/install.sh | bash

# First build
fz build -o my_program

# Watch the speed difference
touch src/main.c && fz build -o my_program

Follow updates: https://t.me/wienton/176

The Statement

ForgeZero isn't an improvement on Ninja. It's not an optimization of Make. It's a complete rewrite of what a build system should be, informed by:

Modern hardware (multi-core, SIMD, large memory)
Modern languages (Go, with its goroutines and concurrency primitives)
Real systems work (assembly, bare-metal, cross-compilation)
Security best practices (reproducible builds, SBOM, SAST)

This is a challenge to the industry: Slow builds aren't inevitable. They're architectural choices. Fix the architecture, and speed fixes itself.

If you're tired of waiting for builds. If you're exhausted by CMake configuration. If you just want to code, not configure—ForgeZero is for you.

The Details

Author: Alex Voste

Repository: https://github.com/forgezero-cli/forgezero

Latest Release: v5.3.0

License: GPLv3 (fully open source, fully free)

Fast builds aren't a luxury. They're an engineering necessity. ForgeZero delivers.

Make it so.

Demystifying the Ghost Failure in Firebase Hosting GitHub Actions

Thu, 02 Jul 2026 05:48:04 +0000

Imagine your pipeline throwing a red error badge, yet your production application reflects fresh updates perfectly. This exact scenario confuses countless developers utilizing the official Firebase Extended GitHub Action. The workflow crashes with a cryptic warning message stating that the supplied version is the current active version, casting doubt on your continuous deployment pipeline reliability.

Redundant Release Triggers:

The pipeline triggers file hashing and bundles content assets accurately, uploading them to Firebase storage infrastructure seamlessly.
Idempotent State Conflicts: When consecutive builds contain zero file modifications, Firebase optimization logic flags the release bundle as structurally identical to the live production build.
Aggressive Exit Code Flags: The deployment utility interprets this identical state as an operational failure, returning a non-zero exit status that forcefully breaks your workflow chain.

🛠️ The Modern Implementation Strategy:

Conditional Deployment Guards: Introduce a dynamic file change detection mechanism directly inside your continuous integration step definitions.
Idempotent Workflow Controls: Implement custom workflow scripts leveraging the core Firebase CLI directly instead of using abstract, rigid pre-built actions.
Graceful Exit Optimization: Append standard bash execution overrides to capture identical version notices and rewrite exit statuses cleanly to neutral status levels.

🔗 visit our official site: www.nextbigcreative.com

I Spent 40 Minutes at 11pm Debugging a Deploy That Wasn't Broken

Thu, 02 Jul 2026 03:27:52 +0000

Level: Intermediate · Time: 6 min · Outcome: A deploy that waits for its dependencies instead of racing them

I once spent forty minutes at eleven at night debugging a deploy that wasn't broken. The release script ran the database migration, the migration threw connection refused, the script exited non-zero, the deploy rolled itself back, and I got paged.

So I did the things you do. I read the migration. I read the logs. I checked the database — it was up, it was healthy, it accepted my connection instantly. I re-ran the deploy and it worked. I chalked it up to gremlins and went to bed, which is the part I'm not proud of, because it happened again two days later. That time I watched the timing: the script brought up a fresh database container and started the migration about six seconds before Postgres finished initializing and began accepting connections. The migration was racing the database's boot. Most of the time it won. The times it lost, I lost forty minutes.

The script wasn't wrong about anything except one assumption: that a dependency is ready the instant you ask for it.

In production, dependencies are eventually ready

That's the mental model shift. Networks blip. A service you call returns a 503 for the two seconds it takes to finish a rolling restart. An API rate-limits you with a 429 it fully expects you to retry. A fresh container's database isn't accepting connections for its first few seconds. Treating the first failure as fatal turns every one of these normal, transient conditions into a paged engineer — and the script that handles them isn't smarter — it declines to give up on the first try.

But retrying naively is its own trap. Retry instantly and you hammer a recovering service into staying down. Retry forever and a genuinely dead dependency hangs your script indefinitely. Retry a 404 and you wait a minute to confirm what you already knew. Good retries are bounded, backed off, and selective.

A retry function you can reuse anywhere

#!/bin/bash
# Purpose: survive transient failures instead of dying on the first error
set -euo pipefail

CHECK="✓"
CROSS="✗"

# retry <max_attempts> <command> [args...]
retry() {
    local max_attempts="$1"; shift
    local attempt=1
    local delay=1            # base delay — doubles each round
    local max_delay=30       # cap so the backoff never runs away

    until "$@"; do
        if (( attempt >= max_attempts )); then
            echo "$CROSS '$*' failed after $attempt attempts" >&2
            return 1
        fi
        # 0–2s of jitter so parallel callers don't all retry on the same beat
        local pause=$(( delay + RANDOM % 3 ))
        echo "$CROSS attempt $attempt failed — retrying in ${pause}s" >&2
        sleep "$pause"
        attempt=$(( attempt + 1 ))
        delay=$(( delay * 2 ))
        (( delay > max_delay )) && delay=$max_delay
    done

    echo "$CHECK '$*' succeeded on attempt $attempt"
}

# The actual fix for my 11pm deploy: wait for Postgres to accept connections.
retry 6 nc -z -w 2 db.internal 5432
echo "$CHECK database reachable — running migration"

The whole engine is until "$@"; do ... done. until runs the command and executes the loop body only when it fails, exiting the instant it succeeds. Passing the command as "$@" (after shift-ing past the attempt count) means the function retries anything — a curl, an ssh, a port check, your own script — without caring what it is.

The backoff is the three lines at the bottom of the loop: sleep for the current delay plus a little jitter, then double the delay, capped at max_delay. That gives you 1s, 2s, 4s, 8s, 16s, 30s, 30s…

The jitter is not decoration

RANDOM % 3 looks trivial, and it's the line people delete to "clean up." Don't. Without jitter, a fleet of machines that all failed at the same instant — because the same service went down — will all retry at the same instant, and the same instant after that, producing a synchronized thundering herd that knocks the recovering service straight back over on every round. A few hundred milliseconds of randomness per client spreads the retries out so the service actually gets room to recover. At one machine it does nothing; at fifty it's the difference between recovery and a retry storm.

The mistake that makes retries dangerous

# Good: a transient failure that retrying can fix
retry 6 nc -z -w 2 db.internal 5432

# Bad: retrying a deterministic failure just delays the error 30 seconds
retry 6 curl -fsS https://api.example.com/v1/thing-that-returns-404

A retry loop is only as smart as what you point it at. A port check belongs in a loop because the answer changes — "no" until the database boots, then "yes." A request that returns 404 returns 404 on attempt one and attempt six; the loop just postpones the failure and buries the real status under retry noise. Retry transient failures — timeouts, connection-refused, 429, 5xx, DNS hiccups. Don't retry deterministic ones — a 404, a 401, a syntax error, a missing file. When you can, branch on the exit code or HTTP status and loop only on the codes worth looping on.

For plain curl, its built-in --retry 5 --retry-delay 2 does most of this and is simpler; reach for the function when the thing you're retrying isn't curl, or when you want one backoff policy across a database probe, an ssh call, and a download at once.

Back to 11pm

That deploy never paged me again once the migration waited for the port instead of assuming it. The database still took its six seconds to boot, the network still blipped occasionally — retrying didn't make the dependencies faster. It stopped a normal, transient slowness from being treated as a fatal error, which is most of what "production-ready" means for a script.

Full function with the wait-for-port pattern and the guidance on which failures to retry: https://bashsnippets.xyz/snippets/bash-retry-with-backoff

Retries are the third leg of an unattended job: flock stops overlap, timeout stops hangs, retry rides out the blip. The Hardened Cron Wrapper Generator wires all three into one wrapper, Bash Scripts That Survive Cron is the end-to-end version, and the rest of the library is at https://bashsnippets.xyz

Originally published at bashsnippets.xyz

Building Quantinium: A Focus on Production-Ready AI Systems

Wed, 01 Jul 2026 15:51:29 +0000

I recently worked on Quantinium (https://quantinium.uk), a project focused on building structured, production-ready AI systems with a strong emphasis on reliability and system design.

The core experience reinforced a key principle: real-world AI engineering is less about model experimentation and more about system architecture. I focused on designing stable pipelines, structured data flow, and predictable outputs instead of relying on prompt-driven behavior.

Key work included building modular AI workflows, enforcing output consistency, and optimizing backend integration for scalable performance.

This project strengthened my approach to building AI systems that prioritize clarity, stability, and real-world usability over complexity.

The Pipeline Was Green for Three Weeks. It Had Been Shipping a Build That Never Compiled.

Fri, 12 Jun 2026 16:25:56 +0000

For three weeks a deployment pipeline reported every step green and shipped a build that had failed to compile on every single run. The build step ended in npm run build | tee build.log so the output could be archived. That pipe is the whole story: bash returns the exit status of the last command in a pipeline, which was tee, and tee always succeeds at copying text. The compiler's non-zero exit got thrown away the instant the pipe handed off. The error was sitting right there in build.log. GitHub Actions saw exit code 0, painted the step green, and deployed the broken artifact. Nobody read the log, because the checkmark said there was nothing to read.

That's the defining property of bash in CI, and it's why I treat pipeline scripts differently from anything I run in a terminal: a silent failure can present as success. On a server you watch a command fail in front of you. In a pipeline, a swallowed exit code produces a green checkmark over broken code, and the gap between "the logs show an error" and "the pipeline reports failure" is exactly where outages are born. I wrote the full guide because I've now been burned by every variation of this, and there's a consistent set of habits that close the gap.

There are four failure modes that are specific to CI and barely ever bite you at an interactive prompt. Exit codes swallowed by a pipe — the story above, any command | tee, command | grep, command | sort. Shell provisioning differences — ubuntu-latest gives you bash 5.x, macos-latest gives you bash 3.2 from 2007, and a script using associative arrays or ${var,,} passes on one runner and throws a syntax error on the other in the same workflow. Environment variable gaps — CI sets variables you don't control and omits ones you assume exist, and without set -u a missing $DEPLOY_TARGET becomes an empty string and does something quietly wrong. Interactive-shell assumptions — CI runs a non-interactive, non-login shell that never sources your .bashrc, so a command that works when you type it dies with command not found because the thing that defined it was never loaded.

The header that closes most of these is short, and every line earns its place:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

set -e exits the moment any command fails, so the step exits non-zero and the workflow actually registers a failure. set -u treats an unset variable as an error, so a typo'd $DPLOY_TARGET dies immediately instead of expanding to nothing and corrupting a path. set -o pipefail makes a pipeline return the first non-zero exit among its commands rather than only the last — that one flag is the direct fix for the | tee bug that ran green for three weeks.

Secrets deserve their own paragraph because CI hands you env: values and secrets: values identically — the shell can't tell them apart, the only difference is that Actions masks the secret's literal string in the log. The trap is that the moment you transform a secret (base64-decode it, slice it, interpolate it), the transformed value no longer matches the mask and prints in clear text. Validate required values up front with ${VAR:?} so a missing secret fails at startup with a clear message instead of on line 47 with a cryptic permission denied, and be very careful with set -x in any step that touches a secret.

The pipe-exit-code problem is worth one concrete tool beyond pipefail: PIPESTATUS is an array holding the exit code of every command in the last pipeline, read immediately after:

npm run build | tee build.log
build_rc="${PIPESTATUS[0]}"   # npm's code, not tee's
[[ "$build_rc" -eq 0 ]] || { echo "build failed: $build_rc" >&2; exit "$build_rc"; }

pipefail has one well-known false positive — grep returns 1 when it finds no matches, which is often fine, and under pipefail plus set -e that aborts the script. Absorb it deliberately with || true only where a non-match is genuinely acceptable, and nowhere else, because blanketing every command in || true just reinvents the silent-success problem you're trying to kill.

Docker has its own landmine: every entrypoint script must end with exec "$@". Without it, your script stays PID 1 and your app runs as a child, so when the orchestrator sends SIGTERM on docker stop or a rolling deploy, the signal hits the script, which doesn't forward it, and after the grace period the orchestrator escalates to SIGKILL — abrupt termination, dropped connections, lost in-flight work. exec "$@" replaces the shell with your app so it becomes PID 1 and receives signals directly. The guide pairs this with a wait_for dependency-check pattern and a trap, and the Bash trap & Signal Handler Builder generates the exact signal block an entrypoint needs.

Deploys get the same treatment: deploy into a fresh timestamped directory, flip a current symlink atomically with ln -sfn so traffic never sees a half-written release, keep the last several releases so rollback is just re-pointing the symlink, run a health check after the swap and fail the deploy if it doesn't pass, and stamp the git SHA into the release so "what's running right now" always has an answer. And when a step fails and the logs won't say why, set -x around just the suspect section shows you each command with its variables expanded — a doubled slash or an empty segment in the trace is usually your bug standing in plain sight.

The full guide is the field manual version of all of this — the four failure modes, the safe header, secret validation with a validate_env function, PIPESTATUS and pipefail, Docker entrypoints, the atomic-symlink deploy script with rollback and health check, debugging with set -x, and a production-ready checklist at the end: https://bashsnippets.xyz/guides/bash-scripting-for-ci-cd-pipelines

If your pipeline scripts are dying on the small stuff first — unquoted loops, bad argument parsing, missing traps — the snippet library that feeds into this guide is at https://bashsnippets.xyz

What performance checks belong in CI (and what belongs in monitoring)

Tue, 09 Jun 2026 21:32:39 +0000

Lighthouse CI can block merges or become a job teams ignore. Here is how we split pre-merge checks, bundle limits, and scheduled PageSpeed monitoring so CI only tests what it can do reliably.

A pull request passed Lighthouse CI on Friday and the preview URL looked fine, but on Monday scheduled monitoring flagged checkout INP on the same release.

The merge was not obviously wrong. The pipeline had checked the homepage and a marketing template. The regression was on a route the workflow never tested. That gap is normal once you track more than a handful of URLs, which is why we stopped using CI as a stand-in for production monitoring and limited it to a small set of pre-merge checks.

Below is what we put in the pipeline, what we keep in scheduled PageSpeed monitoring, and what we removed because it only produced failed builds that teams stopped trusting.

When a green CI build still breaks production webperf

CI answers one question: did this candidate build cross the thresholds we set on the URLs we test in a lab environment?

Production monitoring answers another: are the URLs clients care about still inside budget after deploy, next week, and on templates we never added to the workflow file?

The problems we see most often:

CI tests the wrong URLs: it guards / and /pricing while commerce runs on /checkout and /cart.
Preview differs from production: CDN rules, auth, feature flags, or third-party scripts differ between preview hosts and live domains.
Unstable runs: LCP and INP swing run to run on cold CI runners; strict errors on metrics that vary run to run train teams to ignore the job.
Too much in one pipeline: the job tries to enforce field CrUX, SEO rank proxies, or twelve templates on every docs-only PR.

The fix is not more Lighthouse runs on every push. It is choosing checks CI can run reliably and moving the rest to monitoring, with a written rule for who tests what after deploy.

Performance budget CI gates: what we block on merge

We treat performance budget CI gates as hard stops on merge or deploy preview, not as a report for executives.

Block (error) when the check is stable and linked to what the PR changed:

CLS and layout stability on the templates in scope (lab, but repeatable).
Total blocking time or max potential FID proxies when the change touches main-thread JavaScript.
Resource summary limits when bundle size is the likely regression (script bytes, third-party count).
Performance category floor only when the team has stable reference scores on the preview host; otherwise it becomes a warning first.

Warn when the metric is useful but unstable until preview scores stabilise:

LCP on preview URLs with ads, A/B snippets, or cold CDN edges.
INP unless the PR touches interaction-heavy UI and you run enough repetitions to trust the median.

Do not gate in CI (monitor instead):

Field CrUX or Search Console exports (belong after deploy, not in CI).
Rank or traffic proxies.
Every template in the sitemap on every commit.

Copy the numbers from the same policy doc you use in monitoring. Our Watcher guide on performance budgets for web teams defines the metrics and limits; CI should enforce a subset, not a second spreadsheet.

Resource and bundle budgets: what we enforce before Lighthouse runs

Lab scores can stay green while JavaScript weight climbs every sprint. We enforce at least one resource check that does not depend on Lighthouse score swings:

Main bundle size ceiling (webpack performance, size-limit, or bundlesize).
Dependency diff review when a PR adds more than an agreed chunk of kilobytes.
Image budget on static imports when LCP regressions usually trace to hero assets.

Resource gates are fast, deterministic, and easy to explain in a PR comment ("main bundle +42 KB"). They catch the regressions developers actually introduce. Lighthouse catches how those bytes behave on a throttled mobile profile.

Order matters: run fast checks first. There is little value in a five-minute Lighthouse run when a bundle guard would have stopped the merge in thirty seconds.

Lighthouse CI vs scheduled monitoring: split the pipeline jobs

Run Lighthouse CI on the path that produces a candidate URL: build, preview deploy, health check, then lhci autorun (or collect + assert). Run it on PRs to protected branches and on release tags, with path filters so markdown-only changes do not run Lighthouse unless you want them to.

Run scheduled PageSpeed monitoring after merge: production or staging URLs, client URL groups, a test schedule you can state in a client report, alerts when budgets break in production.

We document the split in onboarding:

Phase	Owner	Question
Pre-merge	CI (LHCI + bundle guards)	Did this branch break budgets on the URLs we test in preview?
Post-deploy	Monitoring product	Are production templates still inside budget on the schedule we promised?

Use the same thresholds in both places. If CI and monitoring use different numbers, engineers ignore both.

For wiring LHCI, assertions, and GitHub Actions patterns, the step-by-step lives on our blog: How to set up performance budgets in CI/CD pipelines. This post is the decision guide before you paste YAML.

Performance budgets in CI for agency client repositories

Agencies rarely maintain one pipeline. Client A is Shopify with theme checks; Client B is Next.js on Vercel; Client C still ships WordPress with a plugin stack nobody wants to touch in CI.

Our default agency CI template is intentionally small:

One or two preview URLs the client agrees are highest risk (often homepage plus one revenue template; two LHCI URLs max on small retainers).
One bundle or size guard when the stack exposes a build artefact.
Written rule in the README: after deploy, monitoring watches the full URL list the SOW names.

We do not paste the client's full performance budget table into every repo. Procurement and account teams already struggle when monitoring scope is vague; CI scope should be narrower and written in the README ("LHCI runs on these two preview URLs; full template list is in Watcher").

Multi-tenant monitoring tracks many sites over time. CI blocks merges when preview tests show a clear regression on the paths in scope.

CI webperf gates we moved to warnings or removed

After several quarters when the CI job failed too often, we changed policy:

Removed full-site crawls inside CI (flaky, slow, duplicates monitoring).
Removed failing PRs on single-run LCP without numberOfRuns ≥ 2.
Moved performance score on preview to warn until preview parity with production was documented.
Moved INP on long-tail templates and logged-in routes to monitoring only unless the PR touches that flow.
Kept CLS regressions on in-scope templates and bundle ceilings on main entry points as errors.

When the main branch fails every other day, teams bypass the job. Fewer error rules that people act on work better than dozens of metrics.

Internal checklist before adding another CI gate

Before we add a new webperf step to a client pipeline, we ask:

Does this metric fail for reasons the developer on the PR can fix?
Is the preview URL stable enough to trust an error (not just a warn)?
Is the same threshold already monitored post-deploy, or will CI create a second set of numbers?
Will the job finish in time for the team's PR review workflow (under ~10 minutes for small sites)?
Can we explain the failure in one sentence in the PR comment?

If two or more answers are no, leave that check in monitoring or in a weekly report, not in merge blocking.

Next step: pick three gates for one repository

Open the client repo where CI fails most often, or your own product site. Write three lines in the README:

URLs Lighthouse CI tests on preview (max two to start).
Errors: CLS, bundle KB, or TBT; pick one lab metric you trust.
Warnings: LCP or performance score until preview scores stay stable.

Implement those before adding a fourth check. Pair with the same numeric budgets you use in monitoring so "green in CI" and "green in the report" mean the same thing.

If you need assertion keys, lighthouserc shapes, and a GitHub Actions sketch, use performance budgets in CI/CD pipelines. If you need the full budget strategy and metric tables first, start with the complete guide to performance budgets.

CI should block merges you would have rolled back anyway. Run everything else on the full URL list in scheduled monitoring, without making every PR wait on Chrome.

Originally published on Hashnode.

From Local Tests to CI in One Command

Tue, 12 May 2026 13:46:58 +0000

Why Set Up CI Early

Once you have tests worth running, the natural next step is making sure they run on every push. The sooner you do it, the fewer surprises down the road — things like env vars that only exist locally or paths that work on your machine but nowhere else.

If you've been following this series, you just ran /twd to write your first batch of tests. This is the right moment to wire them into CI, while everything is fresh.

What the Skill Actually Does

/twd:ci-setup is part of the TWD AI plugin. Like /twd:setup, it starts with a discovery phase — detecting your project configuration before asking you anything.

The skill inspects your project to figure out:

Your framework and build tooling
The dev server port your app runs on
Your base path, if you have one configured
Whether you have coverage already wired up, and if not, whether you want it

Most of this is inferred automatically. The main thing it asks you about is coverage. Once it has the full picture, it generates a GitHub Actions workflow tailored to your project.

Here is an example of what it produces:

name: TWD Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

permissions:
  pull-requests: write  # only needed if using contract-report

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v5

      - uses: actions/setup-node@v5
        with:
          node-version: 24
          cache: npm

      - name: Install dependencies
        run: npm ci

      - name: Install mock service worker
        run: npx twd-js init public --save

      - name: Start dev server
        run: |
          nohup npm run dev > /dev/null 2>&1 &
          npx wait-on http://localhost:5173

      - name: Run TWD tests
        uses: BRIKEV/twd-cli/.github/actions/run@main
        with:
          contract-report: 'true'

The workflow starts the dev server, waits for it to be ready, and runs your TWD tests using the official twd-cli GitHub Action. That action handles Puppeteer setup, Chrome installation, and optionally posts a contract validation report as a PR comment. If your app runs on a different port or base path, the skill adjusts the configuration accordingly.

Running It

After you've written tests with /twd, run:

/twd:ci-setup

The skill will work through the detection phase, ask whether you want coverage reporting, then drop the workflow file into .github/workflows/. Push the branch, open a pull request, and your tests run.

That first CI run tends to surface the usual suspects: an env var that exists locally but not in CI, a path assumption that only works on your machine, or a dependency that wasn't properly declared. The skill takes care of the pipeline setup — and CI itself takes care of showing you what was missing.

The Pattern This Follows

If you used /twd:setup to initialize TWD in your project, this feels familiar. Detection first, minimal questions, concrete output. The philosophy is the same: don't make developers manage configuration when the tooling can figure it out.

The two commands are designed to be used in sequence:

/twd — write tests for your components and pages
/twd:ci-setup — make sure those tests run automatically on every push

Neither requires you to know the internals of TWD, GitHub Actions syntax, or coverage tooling. You describe what you want; the skill handles the rest.

What Comes Next

Setting up CI is about making sure your tests run. The next question is a harder one: are you testing the right things? You might have solid coverage on the happy path and nothing on the edge cases that actually break in production.

The next article in this series covers the TWD Test Gap Analysis skill — how it analyzes your existing tests, identifies what is missing, and suggests what to write next. Not just "you have low coverage on this file", but a concrete read of what user flows and failure modes are not being tested.

If you want to follow along or explore the TWD plugin: https://github.com/BRIKEV/twd-ai

Connecting an AWS EKS Argo CD instance to Octopus Cloud

Tue, 12 May 2026 09:45:45 +0000

Argo CD has become the go-to tool for adopting GitOps in Kubernetes environments. Here at Octopus Deploy, we are huge fans of it; in fact, we're lead maintainers of the Argo project.

In a previous article, we looked at connecting an Argo CD instance running on a Self-Hosted Azure Kubernetes Service (AKS) to Octopus Deploy. While this was neat, AWS still leads the cloud infrastructure market at around 29% as of Q3 2025, according to Synergy Research Group, so if you have been wondering how to replicate that tutorial on EKS, you're in the right place.

Why connect Argo CDs on EKS to Octopus Cloud?

Running Argo CD on EKS gives you GitOps-driven deployments into Kubernetes, but Argo CD alone does not cover the full delivery picture. On the other end of that picture is Octopus, which coordinates the release process around it: approvals, environment progression, runbooks, and a single view across Kubernetes and everything else you ship.

Octopus Cloud customers get a mostly plug-and-play experience because Octopus has already configured the communication rules on its side, and the Helm commands generated by the connection wizard handle the rest (assuming no egress restrictions sit in front of the Argo CD cluster).

In our previous blog, a lot of the steps came about as a result of needing to open up certain ports in order to establish connectivity with your Octopus server; the majority of this goes away when you run in AWS and use Octopus Cloud.

Prerequisites

In order to follow along with this portion of the tutorial, you will need the following tools installed locally.

Terraform, for provisioning ArgoCD and the Argo CD Gateway onto the cluster
Helm, for installing the gateway chart
kubectl, for interacting with the cluster
ArgoCD CLI, for generating an auth token for the Octopus gateway account
An Octopus API key, for accessing the Octopus Deploy REST API

Step 1: Provision a Kubernetes cluster

If you already have a cluster, feel free to skip to the next section. However, if you don't, the following lines will use the EKS CLI, to create one.

To follow along, run the following commands:

Create a cluster configuration:

cat << 'EOF' > cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: basic-cluster
  region: eu-north-1

nodeGroups:
  - name: ng-1
    instanceType: m5.large
    desiredCapacity: 10
    volumeSize: 80
    ssh:
      allow: true # will use ~/.ssh/id_rsa.pub as the default ssh key
  - name: ng-2
    instanceType: m5.large
    desiredCapacity: 2
    volumeSize: 100
    ssh:
      publicKeyPath: ~/.ssh/id_rsa.pub
EOF

The cluster configuration above will provision two nodes with m5.large as the instance type, feel free to tweak the configuration as you see fit.

Create the cluster by running:

eksctl create cluster -f cluster.yaml --timeout 45m

NOTE: The --timeout flag here is to ensure the CLI does not timeout before the cluster is done being provisioned.

Finally, you can obtain the Kubeconfig for the cluster created by running:

eksctl utils write-kubeconfig --cluster=basic-cluster --region=eu-north-1

Step 2: Install ArgoCD

The next step is to install ArgoCD. Using the helm chart is the most straightforward means of installing.

Head back to your terminal and run the following command:

helm install argocd argo-cd \
  --repo https://argoproj.github.io/argo-helm \
  --create-namespace \
  --namespace argocd \
  --wait \
  --timeout 10m \
  --values - << 'EOF'
configs:
  cm:
    accounts.octopus: apiKey
  rbac:
    policy.default: "role:readonly"
    policy.csv: |
      g, admin, role:admin
      p, octopus, applications, get, *, allow
      p, octopus, applications, sync, *, allow
      p, octopus, clusters, get, *, allow
      p, octopus, logs, get, */*, allow
EOF

The output is similar to:

NAME: argocd
LAST DEPLOYED: Fri Apr 15 14:32:08 2026
NAMESPACE: argocd
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:

The command will install Argo CD into a namespace called argocd, the extra values file tacked on sets up the account Octopus will use to talk to it.

accounts.octopus: apiKey creates a dedicated octopus user in Argo CD and enables API key authentication for it, so Octopus has its own identity rather than piggybacking on admin.

The rbac block then scopes what that user can do: policy.default: "role:readonly" makes read-only the baseline for any account without an explicit grant, and the policy.csv lines grant the octopus user only the permissions it needs.

The values above become relevant once we install the Octopus Argo gateway.

What is the Octopus Argo gateway?

Simply put, the Octopus Argo gateway is a component required to establish a secure, TLS-encrypted connection between an Argo CD instance (running in a Kubernetes cluster) and the Octopus Deploy.

In contrast to self-hosted setups where you need to create firewall rules and potentially consider what certs to use to secure the communication, the Octopus Argo gateway handles it all.

Generate an authentication token

In order for the gateway to authenticate with the ArgoCD instance you just deployed, an authentication token is required. To create one, run the following commands:

Grab the auto-generated admin password

ARGOCD_PASSWORD=$(kubectl get secret argocd-initial-admin-secret \
  -n argocd \
  -o jsonpath='{.data.password}' | base64 --decode)

Port-forward the Argo CD server to localhost:18080

kubectl port-forward svc/argocd-server -n argocd 18080:443 &
PF_PID=$!
sleep 5

Log in as admin

argocd login localhost:18080 \
  --username admin \
  --password "$ARGOCD_PASSWORD" \
  --insecure \
  --grpc-web

Mint a token for the octopus account

argocd account generate-token \
  --account octopus \
  --insecure \
  --grpc-web

Be sure to copy and save the token.

Clean up the port-forward

kill $PF_PID

Step 3: Link your cluster

From here on out, installing the Octopus Argo gateway is only a few clicks.

Within the sidebar on the left side of your Octopus dashboard, click on the tab Argo CD Instances.

Figure 1: Octopus dashboard displaying the Argo CD Instances sidebar menu option

Within the Argo CD instance tab click on Add Argo Instance.

Figure 2: Screenshot of the Argo Instance button

Up next, you should be greeted with the following menu.

Figure 3: Screenshot showing the Argo CD instance registration form

Enter a name for this cluster's connection, and if you wish you can select an environment for the ArgoCD instance. Leave the DNS name as the default unless you modified it during the Helm install.

Paste in the authentication token you generated earlier and hit next.

Figure 4: Configuring a project to deploy with Argo CD

Select which project you would like to connect your Argo instance to and hit Configure Project.

Right after the configuration wizard, it will return a Helm install command which looks something like this.

helm install --atomic \
  --create-namespace --namespace octo-argo-gateway-eks-dev \
  --version "*.*" \
  --set registration.octopus.name="EKS-dev" \
  --set registration.octopus.serverApiUrl="https://<your-octopus-instance>.octopus.app/" \
  --set registration.octopus.serverAccessToken="<your-octopus-server-access-token>" \
  --set registration.octopus.environments="{development}" \
  --set registration.octopus.spaceId="Spaces-1" \
  --set gateway.octopus.serverGrpcUrl="grpc://<your-octopus-instance>.octopus.app:8443" \
  --set gateway.argocd.serverGrpcUrl="grpc://argocd-server.argocd.svc.cluster.local" \
  --set gateway.argocd.insecure="true" \
  --set gateway.argocd.plaintext="false" \
  --set gateway.argocd.authenticationToken="<your-argocd-token>" \
  eks-dev \
  oci://registry-1.docker.io/octopusdeploy/octopus-argocd-gateway-chart

Paste the command in your terminal and in a couple of seconds your cluster should be connected!

Figure 5: Confirmation of a connected cluster in the Argo CD Instance tab

When do I pick Octopus Cloud over self-hosted?

If you caught our last blog on connecting an AKS cluster to a self-hosted Octopus server, you might be wondering which path fits your setup.

Octopus Cloud makes sense when you want the connection to just work. The wizard generates the Helm command, you paste a token, and you are done. If your team does not have a strong reason to run Octopus on your own infrastructure, this is a reasonable path.

Self-hosted Octopus makes sense when the answer to "can the server live in our VPC?" is "it has to." which usually comes down to data residency rules, existing on-premises infrastructure you need to keep using, or security policies that will not let a third party hold your deployment control plane.

You pay for that control with the networking work in Shawn's post: opening inbound ports, handling internal DNS, dealing with certs signed by your own CA, and configuring load balancers for gRPC. None of it is hard, but it is work.

Wherever your clusters live, Octopus fits

In this post, we walked through connecting an EKS cluster to Octopus Cloud.

We also looked at some of the tradeoffs between doing this in a self-hosted environment versus the cloud.

Whichever path fits your constraints, the outcome is the same. Argo CD keeps doing GitOps. Octopus handles approvals, environment progression, runbooks, and everything outside Kubernetes that you still need to ship.

Spin up a free Octopus Cloud instance and try the Argo CD integration for yourself, or book a demo if you want a walkthrough.

Happy deployments!

7 Signs Your Release Process Is More Fragile Than You Think

Mon, 11 May 2026 06:37:01 +0000

Most of the release processes look good. Pipelines are green, deploys happen all the time, and nothing is on fire. When something breaks, you only see the cracks, and suddenly you’re trying to answer questions that your tooling doesn’t help with.

DORA metrics are speed and stability. But they don't tell you if rollback works, if approvals live outside of Slack, or if the artifact you tested is the one in production. You can hit all four DORA metrics and still be one bad deploy away from a bad day.

Here are seven signs your process is more fragile than you realize. Individually they seem small. But they don't hold up well under pressure together.

1. "Who deployed this?" takes more than a minute to answer

This is asked almost as soon as a serious incident occurs. If you have to go through CI logs, scroll through Slack, or hope someone remembers to answer your questions, your release process has no operational memory.

The data is out there somewhere, but it’s not structured as a release record; it’s scattered across pipeline runs, commit messages, conversations. Running a CI does not capture the actual deployment intent, and so engineers open multiple systems during incident calls to reconstruct the course of events.

2. The rollback lives in a doc, not in the system

A healthy, well-defined rollback scenario is of utmost importance. Most teams talk about having a rollback plan, but hardly anyone’s ever put one to the test during a real incident.

Here’s what really happens: you scramble to find the old tag, kick off a pipeline, cross your fingers that the environment hasn’t changed, and try to remember all those finicky parameters. Every single step is a chance for things to break. As a post-mortem on approval-gated releases said it directly: “rollback instructions existed in documents, but not in executable policy.”

So you end up staring into that gap, right when pressure’s at its worst. And here is where you start to improvise chaotically.

3. Deployments come down to whoever’s got access and enough urgency to use it

Production access until there is an explicit process usually comes down to who can deploy and needs to do it quickly. That feels like efficiency on a normal day. In a risky change it gets murky: was this checked? this the correct artifact? who is responsible for the go decision?

If you’ve got repo access with deploy permissions, merging code is often synonymous with deploying it. That’s not really a decision, that’s just the way things happen when CI is doing the deployments.

From what I can tell this only happens when something has gone wrong and suddenly no one knows exactly who owns the release.

4. Your deployment metrics are good because you’re measuring the wrong thing

Pass/fail is the wrong indicator here. A release that makes it through on the third try after two reruns is not the same as one that went through cleanly the first time.

The rerun ratio is the most overlooked stat by most teams. Successful releases with high rerun rates are symptoms of brittle steps, flaky dependencies, and environment inconsistencies (even when the terminal status is green). One of the most underdiagnosed CI/CD challenges is environment inconsistency, as it doesn’t show up in pass/fail rates until a regression hits production.

If your deployment telemetry does not distinguish between first-run success and ultimate success, you are missing a reliability signal that is important.

5. Staging and production run different artifacts

If you rebuild per environment instead of promoting the same artifact, there’s no guarantee that what passed QA is what ends up in production. Dependencies can resolve differently, configs can drift, and suddenly you’re debugging a bug you’ve never seen in staging.

The fix is simple in theory: build once, promote the same artifact everywhere. But many CI setups rebuild by default, because that’s how the pipeline is wired.

Most of the time, no one notices until a regression makes it obvious.

6. A CI platform outage also freezes your ability to deploy a hotfix

If deployments run through your CI, a platform issue can take out both builds and releases at the same time. The moment you most need to ship a fix is the moment you can’t.

I can say that this isn’t rare. Looking at a set of failed or delayed releases, the root cause was often not bad code but a fragile setup: wrong variables in CI, missing approvals, and unclear artifact versions.

When your release layer depends entirely on your build layer, their availability becomes the same.

7. Deployment knowledge concentrates on one or two people

What I mean is that the team's ability to ship depends on specific individuals being present and available.

You’ll agree that this is the least dramatic sign. Yet I’m convinced it’s the most damaging long-term. Teams usually notice it when the "deployment person" takes a holiday and suddenly nobody is confident releasing anything. Or when someone leaves and takes the full process knowledge with them.

Deployment governance isn't just an operational concern. It's a scaling and team health concern.

Why do these signs appear together?

None of these is a crisis on its own. That’s why they tend to stick around for so long.

Together, they point to a release process optimized for the easy case: normal days, routine deploys, no incidents. The root cause is the same across all seven. CI and CD are different responsibilities. One is about validating code, the other is about governing how it moves. But most setups try to handle both with a single system designed for only one of them.

If you've gone deeper on the CI vs. CD distinction, the pattern becomes clear quickly. The coupling feels efficient early and becomes fragile at scale. Build maintainers end up carrying production risk decisions they don't own. Deployment logic accumulates inside pipelines built for something else.

As one measurement framework puts it: standardization doesn't remove complexity. It makes complexity measurable. And once it's so, you can improve it deliberately instead of reacting to isolated incidents.

How many did you recognize?

At first, you’ll most probably notice one or two. That’s nothing unusual; every team goes through this before major issues show up. But when you start seeing four or five, here’s where cracks are about to show. Things still run, but it’s clear the process is bogged down and getting heavier than it should be. Hit six or seven, and it’s trouble. The next incident is probably going to be costly for your team. Not just from any bug, but because all that extra process is slowing everything to a crawl. The faster you notice this pattern, the better.

What to do about it

The good news is that you don’t have to dismantle your pipeline to fix any of this.

The main change here is splitting up the purpose of CI and the purpose of releasing. CI’s whole job is to build, test, and create an artifact. And it does that just fine. Everything after that (what version goes where, who approved it, and what happens if it fails) is a whole other responsibility that needs its own structure.

So, when you put this in practice, you end up with four key things:

build the artifact once, then push it to each environment instead of rebuilding at every stage
make approvals a real, tracked step, not just a message in Slack
test your rollback process before you face an actual incident
keep your release history somewhere safe, so it doesn’t disappear on your CI failure

You don't actually need a large platform. This thin layer between build and release, which owns the release decision, captures the audit trail and handles rollback as policy rather than improvisation, removing most of the ambiguity without much overhead.

I’ve seen this pattern enough times to know the structure problem only reveals itself under pressure. Everything is fine until it isn't. And when something breaks at a bad time, the question is not whether you can fix the bug, but whether you can understand what happened quickly enough to do something about it.

Beyond CI/CD: Introducing Semantic Validation in DevOps Pipelines

Tue, 05 May 2026 17:11:12 +0000

Most DevOps conversations focus on one question:

Should CI and CD be separated?

That matters—but I think it misses the deeper issue.

Modern pipelines are good at answering:

Can this code build?
Do the tests pass?
Can it be deployed?

But they rarely answer a more important question:

Should this system state exist at all?

This is where many pipelines fail quietly.

A deployment can be technically successful while still introducing:

semantic inconsistencies
invalid state transitions
ambiguous rollback paths

In other words:

«Passing tests does not guarantee system truth.»

So I’ve been exploring a different architecture:

CI → Semantic Validation → CD

Where:

CI handles build and test
Semantic Validation enforces meaning and state integrity
- SKOS for controlled semantics
- RDF for graph projection
- SPARQL for rule enforcement
CD promotes only validated system states

This changes deployment from:

shipping code → promoting validated reality

Benefits:

Clear state traceability
Deterministic rollback
Stronger governance over production changes

Rollback becomes simple:

Replay a known validated state.

Not rebuild-and-hope.

I believe the next evolution of DevOps is not just faster pipelines.

It’s pipelines that can formally answer:

What is allowed to exist in production?

Curious how others think about this.

Do your current pipelines validate only code, or system meaning too?

Do you separate CI and CD?

Tue, 05 May 2026 12:38:19 +0000

In a lot of setups I’ve seen, CI is responsible for everything—build, test, deploy, and even rollback.

I'm convinced that this approach is not that efficient. Even the most basic things become harder than they should be. It’s no longer immediately clear what’s actually running in production, who approved the release, or what the last stable version was. And when something goes wrong, rolling back quickly without rebuilding turns into a stressful, manual process.

The information is scattered across pipeline logs, commits, and chats.

So I'm curious how you handle this. Do you separate CI and CD? If yes, what tools do you use? Or do you keep everything inside pipelines?

Let's discuss!

Octopus Easy Mode: Safely Creating Projects With the AI Assistants

Tue, 05 May 2026 10:16:56 +0000

AI in DevOps faces an interesting problem.

According to Octopus Deploy's AI Pulse report, 44% of developers say they're frustrated with AI outputs that are "almost right, but not quite."

You’ve likely experienced it yourself: a large language model generates a deployment config missing a critical variable, or suggests a step that works in staging but breaks in production. Which is why many engineers will use AI to whip up a script. Fewer will let it configure a production pipeline unsupervised.

So far in the Easy Mode series, we’ve walked through how to use the Octopus AI Assistant for specific, bounded tasks, from adding manual interventions to setting environment-scoped variables.

Each post focused on a task where you could verify the result before it hit production. This post continues that pattern. We’ll cover how to create projects and what to do if you need to rollback a change you are not satisfied with.

What the AI Assistant Does (And What We're Building)

If this is your first easy mode article, it is worth explaining what the Octopus AI assistant is and exactly what we will be building.

The Octopus AI assistant is a Chrome extension that integrates AI directly into Octopus Deploy. It can answer questions about your instance, help you build projects, and guide you through configuration, all without leaving the browser tab you are already in.

A common question users often have is “Does Octopus use my data for training?” to get that out of the way, the extension does NOT train AI models on your data.

Demo: Creating a Project With the AI Assistant

In this walkthrough, we will create a project using the Octopus AI assistant and simulate responding to a request to revert a change.

Prerequisites

In order to follow along with this portion of the tutorial, you will need the following:

An Octopus Cloud account: If you don’t have one, you can sign up for a free trial.
The Octopus AI Assistant Chrome extension: You can install it from the Chrome Web Store.

Upon installation, you should see a new icon in the bottom right.

Figure 1: Visual of the Octopus AI Assistant after Chrome extension installation.

Create a project

With the extension installed, the next step is to create a project.

Paste the following prompt into the Octopus AI Assistant and run it:

Create a Script project called " Git backed Project".

Figure 2: Visual of creating and executing a project in Octopus AI assistant

Upon hitting enter, the Octopus AI assistant will process your request and generate the OpenTofu code required. This is important because using OpenTofu makes your results much more deterministic.

Figure 3: Generating the OpenToFu

Many engineers have expressed doubts when using generative AI, as the results tend to vary. Generating the OpenTofu required to fulfill your requests helps the assistant achieve greater accuracy, as there is only one way to create a project via the OpenTofu provider.

Additionally, if you were to do this manually, it would take anywhere from a couple of minutes to a few minutes, depending on how many projects you intend to create and whether you need to modify an existing OpenTofu module.

Your Safety Net: Version-Controlled Projects

So far, the Octopus AI assistant has created your project. The obvious question is: what if something is wrong?

Thankfully, Octopus Deploy lets you back any project with a Git repository. When you enable this, every change to that project, whether you made it or the assistant did, becomes a Git commit.

To set this up, go to your project settings and connect a Git repository. Once connected, Octopus stores your project configuration in a .octopus folder in that repo. Deployment processes, variables, triggers, all of it lives in version-controlled files.

Figure 4: Visual of version control being enabled in project settings

Now look at what happens after the AI assistant creates your project. Open the Git history for the repo. You'll see the commits the assistant generated, and you can inspect exactly what it configured.

Figure 5: Visual image of the committed files

Clicking into any commit gives you the full diff, and if something looks off, you have two options: fix it in the next commit or revert to a known-good state. This is the same workflow you already use for application code.

Figure 6: Visual image of the diffs

This matters because you can use the assistant with more confidence, because changes are auditable, and because relying on a config-as-code approach means there is only one way to do things.

Tips for Getting the Most Out of AI-Assisted Setup

At this stage, if you’re looking to try out an AI-Assisted workflow, here are some quick tips on how to get better returns on your new workflow:

Write short, specific prompts: The more detail you pack into a single prompt, the longer the assistant takes to process it. Keep prompts scoped to a single task.
Include platform and environment details. The assistant produces better results when you tell it what you're deploying to. Specify the target (Kubernetes, Azure App Service, AWS ECS), name your environments, and indicate your preferred deployment strategy (rolling, blue/green, canary) if applicable.
Plan to customize after generation: The assistant handles straightforward project scaffolding well. Complex variable scoping, multi-tenant configurations, and custom scripts are better handled manually after the initial setup. Use the AI to get the structure in place, then fine-tune. Git tracks everything either way.

Beyond project creation, the AI Assistant can also help with troubleshooting failed deployments and surfacing best-practice recommendations.

Build Fast, Ship Safe

In this post, we used the Octopus AI Assistant to create a project from a prompt and then backed it with Git via Config as Code.

While the conversation around AI in DevOps evolves, having the ability to use it on your own terms is what makes the Octopus AI assistant a good choice for teams looking to dip their toes in the water.

If you want to try this yourself, install the Octopus AI Assistant Chrome extension and give it a shot.

Happy deployments!

A Soft Introduction to EasyShell

Tue, 05 May 2026 02:31:45 +0000

"Can it be simpler?" - A recurring thought I've had while working on build automation.

EasyShell (command: easy) is a minimal shell scripting language designed from the ground up for simplicity and cross-platform build/process automation. If you’ve ever felt that PowerShell’s syntax is powerful but easy to forget, or that traditional shell scripts get messy across operating systems, EasyShell aims to strike a cleaner balance.

It's implemented in C# and runs on .NET, giving you natural access to the entire .NET ecosystem while keeping the scripting experience lightweight and approachable.

Design Principles

One command per line (shell-like flow)
Very small set of core concepts
Strongly-typed global variables
Seamless .NET interop via reflection
Expressions in parentheses for inline evaluation
Easy to read, write, and remember

Version 0.1.0 is now available — functional for everyday automation tasks.

Core Language Features

Variables

All variables are declared with an explicit type and live in global scope:

INTVAR Count 42
BOOLVAR Debug TRUE
DOUBLEVAR Ratio 3.14
STRINGVAR Message "Hello from EasyShell"
HANDLEVAR Now System.DateTime.Now   # stores object instances

Variable names are case-insensitive
Reference with $VarName
Re-assign with $VarName = value (or expressions)

Expressions (The Power Feature)

Wrap sub-commands in parentheses to evaluate them inline:

STRINGVAR Formatted (System.String.Format "Count = {0}" $Count)
BOOLVAR IsBig (> $Count 100)
STRINGVAR Today (format "{0:yyyy-MM-dd}" (GetDate))

Calling .NET

Static members (fully qualified):

System.Console.WriteLine "Hello World!"
System.IO.File.WriteAllText "output.txt" $Message

Instance methods via HANDLE:

HANDLEVAR Now System.DateTime.Now
STRINGVAR Stamp (CALL $Now ToString)

Control Flow

IF (== $Count 42)
    print "The answer!"
ELSEIF (> $Count 50)
    print "Too high"
ELSE
    print "Something else"
END

INTVAR I 0
WHILE (< $I 5)
    System.Console.WriteLine $I
    $I = (+ $I 1)   # arithmetic helpers coming soon
END

Functions

Simple named blocks (no parameters — use globals):

FUNC BuildProject
    print "Building..."
    dotnet build
END

CALL BuildProject

External Commands

Any command that isn’t a keyword or .NET call falls back to executing an external process (with stdout capture):

git status
dotnet publish ...

Real-World Example

Check out the full build script used by the project itself:
BuildEasyShell.easy

It demonstrates:

Path handling and folder creation
Running dotnet publish
Conditional logic for OS/architecture
Creating a dated zip package

This script is self-contained and shows how EasyShell can manage non-trivial automation tasks.

Quick Start

# REPL mode
easy

# Run a script
easy path/to/script.easy

# Help
easy --help

A dedicated VS Code syntax highlighting extension is available: Methodox.easyshell

Who Is It For?

EasyShell is ideal for:

Build automation scripts
Simple CI/CD glue
Daily developer tooling
Anyone who wants .NET power without heavy dependency

It's intentionally minimal — no advanced Bash-style features, just the essentials done in a way that’s hard to forget.

Give it a try and let me know what you think!

DevLog 20251214: Yet Another Shell Script – Keeping Build Automation Simple with EasyShell

Tue, 05 May 2026 02:23:48 +0000

As we roll out more tools, both internal and public, there has been a growing need for easy-to-maintain shell scripting and build automation. Build automation is complex enough without all the syntax burden - my favorite is to use PowerShell, but often it feels like wrestling with the tool itself. PowerShell is powerful, sure, but its depth often means hunting down syntax I know I've used a dozen times before. Bash has its quirks across platforms, and honestly I don't know good ways of using it outside MinGW, Git Bash, or WSL - none of which felt quite "right".

I wanted something lighter: a shell scripting tool that's easy to pick up, cross-platform, and hard to forget once you start using it.

So I made EasyShell (or just easy): a minimal shell scripting language implemented in C# that focuses on process automation with a very small set of concepts.

Motivation and Design

EasyShell is for "lazy" developers (like me) who just want to automate common tasks - calling .NET APIs, running external tools, handling simple logic - without a steep syntax curve or platform-specific headaches. It combines a shell-like one-command-per-line flow with strong typing and easy .NET interop. No advanced Bash-style wizardry; just straightforward scripting.

It could also serve as a nice companion alongside other tools in the long-term Divooka ecosystem.

Quick Taste of the Syntax

Files typically use the .easy extension, with syntax highlighting available via the VS Code extension.

Here's a small sample from the repo:

# Comments start with #
HANDLEVAR NOW System.DateTime.Now

# Instance method via CALL
STRINGVAR DATE (CALL $NOW ToString)

# Static .NET call with expression
STRINGVAR VALUE (System.String.Format "Current time: {0}" $DATE)

STRINGVAR PATH "C:/temp/greeting.txt"
System.IO.File.WriteAllText $PATH $VALUE

Variables are declared with their type (INTVAR, STRINGVAR, BOOLVAR, DOUBLEVAR, HANDLEVAR), names are case-insensitive, and everything lives in global scope for simplicity. Expressions are wrapped in parentheses for inline evaluation, like (== $X 10) or sub-commands.

Control flow sticks to the basics:

IF ... ELSEIF ... ELSE ... END
WHILE ... END
Simple FUNC ... END / CALL (globals make "returns" easy via variables)

You can call external executables naturally, and non-built-in commands fall through to process execution (stdout capture included).

Early Days

This is version 0.1.0 territory - functional for everyday build tasks, with room to grow (arithmetic helpers, more examples, etc.). You can run it in REPL mode (easy) or point it at a script. Builds and packages are straightforward with the included scripts.

If you've ever wished for something between a full PowerShell script and a quick batch file that "just works" across machines, maybe give EasyShell a try. The GitHub repo has the full language overview, examples, and build details: MethodoxTech/EasyShell.

Orbnetes — Release Orchestration

Thu, 30 Apr 2026 06:45:23 +0000

I kept running into the same problem. CI tools are great at building code, but not at releasing it. Approvals feel bolted on, rollback is improvised, and when something breaks, nobody can answer who deployed what or when.

So I built Orbnetes — a self-hosted release orchestration layer that sits on top of your existing CI and handles everything after the build finishes.

How it works

You connect it to GitHub, GitLab, or your own storage. It picks up the artifact, validates it, and manages the release from there — approvals, environment promotion, rollback, full audit record.

Deployments run through lightweight cross-platform agents (~2MB) inside your own infrastructure. Deployment logic lives in YAML blueprints with a syntax close to GitHub Actions — familiar but scoped purely to release control.

Every release gets a structured record. If something fails, rollback executes against the already-validated artifact not a pipeline re-run.

Key features

Cross-platform agents for Linux, macOS, Windows
YAML blueprints with approval gates and rollback policies
Full audit trail independent of your CI platform
Self-hosted, free, no Kubernetes required

Orbnetes is free to try. Would love to hear your feedback!

CI and CD Are Not the Same Thing: How to Build a Resilient Release Process

Sat, 25 Apr 2026 11:37:27 +0000

If you’re building a software product, your deployment process matters from day one. It’s not a “later” problem or just DevOps work in the background. It directly impacts how stable your product is, how fast you recover from issues, and how confident your team feels when shipping changes.

Most teams know this. But in practice, their release process looks very different.

Instead of a clear system, they end up with a patchwork of CI jobs — a deploy step here, a Slack alert there, a quick workaround that quietly becomes permanent.

It works. Until it doesn’t.

And when something breaks, the same question comes up: what just happened?

If that sounds familiar, this article is for you.

CI and CD are not the same thing

I’ve been building tech products for over ten years now, managing development teams through the full release cycle, from early-stage pipelines to production systems under real operational pressure. That experience made the gap between CI and CD for me impossible to ignore.

Usually, teams treat CI platforms as the default place to do everything: build, test, package, deploy, promote, and sometimes even roll back. At first it feels efficient. Your source code is already in GitHub or GitLab, your pipeline definitions live next to your code, your runners are already there. So why not push deployment and release orchestration into the same system too?

Because convenience at the beginning often becomes coupling at scale.

CI and CD are related, but they are not the same responsibility. CI is about validating change – build the code, run the tests, produce an artifact, confirm the commit is healthy. It answers one question: "Is this change valid?"

CD is about controlled change propagation – selecting a release candidate, deciding where it goes, defining an approval flow, promoting through environments, enforcing role-based access, capturing audit history, handling rollback safely. It answers a completely different question: "Who deployed which version, where, when, and what happened next?"

Those are different questions and they clearly deserve different systems.

The hidden cost of using CI for releases

Using GitHub Actions or GitLab pipelines for deployments is not wrong. It is often the fastest way to get started. But once deployment logic lives inside CI, several problems pile up.

1. Your release process becomes coupled to your source platform

If your CI platform is degraded, your release process is too. GitHub's incident history shows multiple events where Actions jobs failed or were delayed (sometimes in the same window that affected pull requests and API requests).

When your builds, approvals, and production releases all depend on the same platform, one outage freezes both development and delivery simultaneously. If you need to ship a hotfix during that window, "everything in one place" stops looking like an advantage, it becomes a single operational dependency.

2. CI pipelines are great at execution but awkward at release governance

Pipelines are good at running jobs. They are less natural as systems of record for release intent. Yes, you can build approvals into workflows. Yes, you can model environment promotion in YAML. Yes, you can encode rollback logic in scripts. But over time teams start asking questions that CI was never built to answer cleanly:

Which exact version was approved for staging?
Who approved the promotion to production, and when?
Was QA sign-off captured before the release went out?
What was the previous successful version?
Can we roll back automatically if a deployment health check fails?

At that point deployment stops being "just another job" and becomes an operational workflow, one the pipeline was never designed to own.

3. Release logic becomes too tightly bound to repository events

Most CI-driven deployment patterns are built around repository triggers: push to branch, merge to main, create a tag. That works when your release process is simple. But at some point teams start wanting things pipelines were never designed for — build once, deploy many times; promote the same artifact from staging to production without rebuilding it; let operations trigger a deployment without touching the codebase; roll back without re-running anything. The pipeline starts fighting back. That friction is usually the first sign that deployment has outgrown CI.

What this costs at 2am

The real cost becomes visible during a production incident when every minute matters. These are the questions that come up in the first ten minutes:

Who deployed the version currently running in production?
Was QA sign-off captured before this version was promoted?
What was the last known stable version before this deployment?
Can we roll back in the next five minutes without re-running a build?
When CI-driven deployments cannot answer these (because the information is scattered across pipeline logs, Slack threads, and individual memory) the incident takes longer to take care of.

For teams building tech products, every minute of downtime has a direct user-facing impact. And there is a less visible cost too: the team gets slower without understanding why.

Rollback deserves particular attention here. In many CI-driven setups, rollback isn’t a defined operation. It’s a scramble:

find the old tag
re-run a pipeline
hope nothing has drifted
reconstruct whatever parameters were used last time

That’s improvisation under pressure. A release system should treat rollback as a first-class capability – tested, predictable, and fast.

Because at 2am, clarity matters more than anything else.

Keep CI and CD connected, not fused

The fix is not to replace your CI setup. GitHub Actions is powerful, GitLab CI is well-designed, and Jenkins is still deeply useful in many organizations. This is an architectural argument, not a tooling one.

Keep CI where it belongs – building, testing, packaging, publishing a versioned artifact. Let it answer "is this change ready?" and stop there. Then hand off to a separate, independent release layer that owns everything that happens next: which version ships, to which environment, with whose approval, under which rollback policy, with a full audit record that survives independently of your source platform.

That separation gives you something most teams are missing: a controlled release process instead of an automated shell script with side effects.

The best release process is not the one with the fewest steps, it is the one that still makes sense at 2am during a production incident.

How to actually separate CI from CD (and what to use)

If your infrastructure is already mature, there are several established paths.

Spinnaker is a powerful open-source CD platform with strong support for multi-cloud deployments and promotion workflows but it comes with significant operational overhead.

Argo CD is a solid choice for Kubernetes-native teams. It follows a GitOps model and has strong community support, but it is tightly coupled to Kubernetes environments.

Flux takes a similar GitOps approach with a lighter footprint, though it shares the same Kubernetes focus.

And tools like GitLab can handle both CI and deployment in a single platform, something that works well for many teams, especially early on.

But these tools solve different slices of the problem. They also assume a certain level of infrastructure maturity. For many teams, they introduce as much complexity as they remove.

But the key idea is super simple: separate the system that builds artifacts from the one that promotes and deploys them.

That became very obvious to me when I started testing deployment workflows outside of a single-machine setup – running services across isolated environments, with separate infrastructure components, and real failure conditions. The moment you leave the comfort of a local setup, hidden assumptions in pipeline-based deployments start to surface quickly.

That’s the reason the idea of Orbnetes came into my mind. It’s a self-hosted release orchestration layer designed to sit on top of your existing CI, without requiring Kubernetes or heavy operational overhead.

Instead of rebuilding per environment or encoding release logic in pipelines, the system focuses on:

promoting immutable artifacts
defining explicit approval flows
and keeping a release record independent of CI

The goal isn’t to replace your CI, but to give deployments their own control plane – one that still makes sense when things go wrong.

How Orbnetes works in practice

Your CI pipeline stays exactly as it is. It builds, tests, and produces an artifact.

Orbnetes picks up from that point. It integrates with systems like GitHub and GitLab, as well as internal artifact storage or webhooks, and turns build outputs into versioned releases.

Each artifact is validated (for example via checksum) and stored as a release candidate. From that point on, deployments use the same immutable artifact across environments – no rebuilding, no hidden differences between staging and production.

That one decision alone removes an entire class of deployment inconsistencies.

Deployments are executed through lightweight agents running inside your infrastructure. They are cross-platform, isolated, and centrally managed, so environments stay independent, but the release process stays consistent.

Instead of embedding deployment logic in CI pipelines, it lives in blueprints – YAML definitions designed for release orchestration rather than build automation.

name: production-release
inputs:
  target_env:
    type: string
    required: true

jobs:
  backup:
    tags: [linux]
    steps:
      - name: pre-deploy-backup
        run: echo "Backup before deploy"

  deploy:
    needs: [backup]
    tags: [linux]
    steps:
      - name: deploy-release
        run: echo "Deploying artifact"
  verify:
    needs: [deploy]
    tags: [linux]
    steps:
      - name: health-check
        run: echo "Verifying deployment"

The syntax is intentionally familiar, but the responsibility is different: this is about releasing software, not building it.

Blueprints are versioned, reusable, and visible to the whole team rather than hidden inside a pipeline file that only a few people understand.

Before a release reaches a sensitive environment, it goes through explicit approval flows. Roles are separated: the people who build, approve, and deploy are not implicitly the same. Every action is recorded.

If something fails, rollback is not a guess. The system restores the last known good version using the already validated artifact. It is defined upfront, not improvised during an incident.

Every release becomes a tracked event:

what version was deployed
where it went
who approved it
what happened next

And importantly, this record exists independently of your CI platform. If GitHub has an outage, your release history (and your ability to operate) does not disappear with it.

Don't wait for an incident to fix this

CI tells you whether your software is ready. CD determines how it actually reaches production.

Those are different responsibilities and treating them as the same system is where teams start to lose control.

At a small scale, pipeline-driven deployments feel efficient. Everything is in one place, automation is easy, and shipping is fast. But as systems grow, that convenience turns into coupling. And coupling turns into fragility (usually revealed at the worst possible moment).

During an incident, the problem is never just the bug. It’s the lack of clarity:

What exactly is running in production?
Who approved it?
What changed?
How fast can we safely roll back?

The goal is not to add more steps or slow teams down. It’s to make releases predictable, observable, and controlled even under pressure.

That starts with a simple shift: build in CI, release through CD, keep them connected, but not fused.

If this resonates with how your team works, Orbnetes is free to try. The documentation covers full setup in under 30 minutes, and the blueprints library has ready-made examples for common deployment patterns across Linux, Docker, and multi-environment setups.

Remember that your deployment process is part of your product. Treat it that way:)

Why CI/CD Changed Software Development Forever

Fri, 24 Apr 2026 11:13:00 +0000

There was a time when deploying software was scary.

Really scary.

Developers would write code for weeks…

Then manually deploy everything at once and pray:

“Hope nothing breaks in production.”

Sometimes it worked.

Sometimes…

Production exploded

That’s exactly the problem CI/CD helped solve.

Let’s break it down simply

Before CI/CD: Releases Were Painful

Earlier, shipping software often looked like this:

Developers wrote code for weeks
Changes piled up
Big release day arrived
Team manually deployed everything

And then:

Unexpected bugs
Broken builds
Failed deployments
Long downtime

Releasing software felt risky.

⚠️ Common Problems Before CI/CD

Without automation:

Testing was often manual
Bugs were caught late
Deployments were error-prone
Rollbacks were messy
“Works on my machine” became a lifestyle

Sound familiar?

What is CI/CD?

CI/CD stands for:

✅ Continuous Integration (CI)

Developers push code frequently.

Every change automatically:

Builds the project
Runs tests
Checks if anything broke

Problems are caught early.

Continuous Delivery / Deployment (CD)

After code passes checks:

It can be:

Prepared for release automatically
Or deployed automatically

Shipping becomes faster and safer.

What Happens in a CI/CD Pipeline?

Typical flow:

Code Push → Build → Test → Security Checks → Deploy

Simple idea.

Powerful impact.

Why This Changed Everything

Before:

❌ Big risky releases

Now:

✅ Small frequent safe releases

That changes software development completely.

⚡ Real Benefits of CI/CD

Faster Development

Developers ship features quicker.

No waiting for huge release cycles.

Better Code Quality

Automated tests catch issues early.

Less broken code reaching users.

Safer Deployments

Every deployment follows a repeatable process.

No random manual mistakes.

Faster Bug Fixes

Critical issue?

Push fix.

Pipeline runs.

Deploy.

Done.

Real-World Example

Imagine updating a payment feature.

Without CI/CD:

Manual tests
Manual deploy
High risk

With CI/CD:

Push code
Tests run automatically
Deploy safely

Huge difference.

Tools You’ve Probably Heard Of

Popular CI/CD tools:

GitHub Actions
GitLab CI/CD
Jenkins
CircleCI

Different tools.

Same mission:

Automate software delivery.

Why Modern Teams Can’t Ignore It

Today apps update constantly.

Think about:

Netflix
Amazon
Spotify

They can’t do manual deployments all day.

CI/CD makes modern scale possible.

⚠️ Is CI/CD Just for Big Companies?

No.

Even small projects benefit.

Because automation early saves pain later.

Huge lesson.

The Big Shift

CI/CD changed development from:

❌ “Deploy carefully and hope”

✅ “Deploy confidently and often”

That’s a mindset shift.

Not just a tool shift.

Final Thought

CI/CD didn’t just make deployments faster.

It changed how software gets built.

Less fear.
More automation.
Faster delivery.

And honestly…

Modern software would look very different without it

How to Set Up Performance Budgets in CI/CD Pipelines

Sun, 19 Apr 2026 23:55:33 +0000

A performance budget in production is a line you refuse to cross. In CI it is the same line, enforced before a merge or deploy lands. Done well, the pipeline fails fast when a change regresses Core Web Vitals proxies, bundle weight, or your own custom thresholds, so you fix it in the branch instead of shipping first.

This guide assumes you already know why budgets matter. For the full conceptual frame and metric tables, read The Complete Guide to Performance Budgets for Web Teams. Here we focus on wiring budgets into CI/CD: tools, config shapes, and operational pitfalls.

What “performance budget in CI” usually means

In pipelines, teams typically enforce lab metrics (Lighthouse scores and timings such as LCP, CLS, INP where available, TBT, and so on) against numeric ceilings or floors; resource budgets (caps on JS, CSS, image bytes, request counts, or third-party weight); or custom checks (synthetic steps that call your own scripts or APIs after a build).

Lighthouse CI is the common open path for lab metrics because it runs Lighthouse in a controlled environment, stores results, and supports assertions against budgets. Pair it with bundle analysers or size limits when regressions come from dependency drift rather than layout alone.

CI budgets are not a substitute for field data or scheduled monitoring across many URLs. They gate the build you are about to ship. Products such as Apogee Watcher focus on ongoing lab schedules, portfolios, and alerts; see How to Set Up Automated PageSpeed Monitoring for Multiple Sites for that workflow.

Before you write YAML: pick URLs and environments

CI runs should be deterministic enough to trust. Preview URLs from Netlify, Vercel, Cloudflare Pages, or internal preview hosts work if the pipeline waits until the deploy is reachable. Local static servers (serve, http-server) suit static sites; SPAs often need the production build and a correct BASE_URL. Auth walls break Lighthouse unless you script login or use a dedicated test route.

Document which URLs represent home, a heavy template, and checkout or app shell if those differ. One URL with a loose budget hides regressions on another; at minimum, list primary templates in lighthouserc or equivalent.

Path 1: Lighthouse CI (LHCI)

Install

Add Lighthouse CI to the repo (dev dependency is typical):

npm install --save-dev @lhci/cli

Minimal `lighthouserc` (assertions + collect)

LHCI reads configuration from lighthouserc.js or lighthouserc.json. Example JSON shape:

{
  "ci": {
    "collect": {
      "url": [
        "https://your-preview.example.com/",
        "https://your-preview.example.com/pricing"
      ],
      "numberOfRuns": 3
    },
    "assert": {
      "assertions": {
        "categories:performance": ["error", { "minScore": 0.9 }],
        "largest-contentful-paint": ["error", { "maxNumericValue": 2500 }],
        "cumulative-layout-shift": ["error", { "maxNumericValue": 0.1 }],
        "total-blocking-time": ["warn", { "maxNumericValue": 300 }]
      }
    },
    "upload": {
      "target": "temporary-public-storage"
    }
  }
}

Notes:

numberOfRuns: multiple runs reduce noise; three is a common starting point.
Assertion keys map to Lighthouse audit IDs; align numeric budgets with your performance budget template and team policy.
upload.target: temporary-public-storage is fine for getting started; teams often move to LHCI server or skip upload in pure gate mode.

Wire the CI job

Invoke LHCI after the app is built and the target URL responds. Typical flow:

Install dependencies.
Build the site (if needed).
Deploy to preview or start a static server in the background.
Wait until the test URLs return HTTP 200.
Run lhci autorun (or lhci collect then lhci assert).

If you use GitHub Actions, a dedicated job with timeout-minutes and a health-check step avoids flaky “site not ready” failures. A minimal pattern is to probe the preview URL before lhci autorun, for example with curl -fsS --retry 5 --retry-delay 5 --retry-connrefused "$PREVIEW_URL". --retry-connrefused matters because a deploy that is not listening yet often returns “connection refused”, which plain curl retries do not treat as transient by default. Store the same base URL in a CI variable and pass it into lighthouserc or environment overrides your setup supports, so you do not duplicate hostnames in three places.

Resource limits: `budgetsFile` vs assertions

Lighthouse CI can assert against a budget.json-style file via assert.budgetsFile (path relative to the working directory). In the upstream configuration model, that budgetsFile mode is an alternative to filling assert.assertions with audit IDs; it is not mixed with other assert options in the same way. Check the Lighthouse CI configuration reference for the exact rules your CLI version supports.

If you want lab metrics (LCP, CLS, and so on) and transfer or request budgets in one assertions block, use Lighthouse CI’s resource-summary assertions (for example resource-summary:script:size, resource-summary:third-party:count) alongside the audit keys. Sizes there use bytes in assertion options; the standalone budget.json format often documents kilobytes, so keep units straight when you copy numbers between files.

Whether you use a checked-in budget file or assertion rows, treat the file like any other policy: generate from your design system pipeline or review diffs in PRs so “max JS kilobytes” does not drift silently.

Path 2: GitHub Actions sketch

Below is a pattern, not a drop-in for every stack: replace build commands, Node version, and URL discovery with your own.

name: Lighthouse CI
on:
  pull_request:
    branches: [main, develop]

jobs:
  lhci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npm run build
      # Example: wait for preview deploy via your provider’s CLI or API, then:
      - run: npx @lhci/cli autorun
        # Optional: set LHCI_GITHUB_APP_TOKEN if you use the Lighthouse CI GitHub App
        # so upload can post PR status checks. Not required for a local pass/fail gate.
        # env:
        #   LHCI_GITHUB_APP_TOKEN: ${{ secrets.LHCI_GITHUB_APP_TOKEN }}

Many teams split build and LHCI across workflows so preview deploy completes first; use workflow_run or provider webhooks if needed. The critical invariant is that Lighthouse runs against the same artifact users will get, not a half-built tree.

On GitLab CI, CircleCI, or Buildkite the same steps apply: install Node, build, wait for URLs, then run npx @lhci/cli autorun (or your package script). Cache node_modules between runs when your runner allows it; cold installs dominate wall time on small changes.

Resource budgets alongside Lighthouse

Lab metrics can pass while JavaScript weight creeps up on every sprint. Add at least one of: bundle size limits (bundlesize, size-limit, or webpack’s performance hints), or dependency reviews in the PR template so someone notices a multi-megabyte new dependency.

Treat resource budgets as complementary to LCP and CLS gates. A slow-LCP fix might add twenty kilobytes and still help users; a green Lighthouse run with a nine-hundred-kilobyte main bundle remains fragile.

Flakiness and false positives

CI environments are colder than a developer laptop. Variance in LCP and CLS is normal: mitigate with multiple runs, pinned Chrome, and stable network throttling settings in LHCI. Third-party ads or A/B scripts can differ run to run; block known domains in a test profile or point at a clean route. A cold CDN edge on the first request after deploy can skew timings; an optional warmup GET before LHCI helps.

If the main branch is red every other day, teams stop trusting the job. Prefer warnings on noisy metrics and errors on stable ones until baselines settle.

How this pairs with product-side budgets and alerts

CI answers whether this change broke your thresholds on the URLs you chose. Scheduled monitoring answers whether you are still inside budget next week across the pages you track in production.

If you already use performance budgets and email alerts in Apogee Watcher, treat CI as the pre-merge gate and Watcher as the continuous check on real site inventories. Same vocabulary, different phase of the lifecycle.

Apogee Watcher and CI: agency API (in active development)

You do not need a vendor API to get value from Watcher next to LHCI. Many teams keep Lighthouse CI (or Lighthouse CLI) in the pipeline for fast feedback on the preview URL, and keep budgets, schedules, and digests in Apogee Watcher so production-facing URLs and client reporting stay in one product. Align the numbers: copy thresholds from your performance budget template into both LHCI assertions and Watcher site budgets so “green in CI” and “green in monitoring” mean the same thing.

Apogee Watcher is building a plan-gated customer HTTP API (under /api/v1) aimed at agency workflows. It is in active development: behaviour, routes, and reference documentation will firm up as releases land. Eligible plans will expose the capabilities below; exact rollout timing will follow release notes.

For agency users, the API will be supporting:

Check test results: read latest (and historical) PageSpeed outcomes for monitored pages without opening the dashboard.
Trigger a new test: request a fresh run after a deploy so CI or a script can wait on Watcher instead of calling the Google PageSpeed API key from your own workers.
Retrieve aggregated reports: pull summary or roll-up reporting suitable for client packs or internal gates.
Retrieve historical trends: chart-friendly series so pipelines or internal tools can compare this build against last week, not only the last run.

Why that matters next to LHCI: once those endpoints are live, a pipeline will be able to trigger a test, poll until results land, then fail if metrics breach the same budgets you set in the app. Quota and retries will stay on Watcher’s side, and the failing run will remain tied to stored results and trends for client conversations, not only a line in your CI log.

Until the public reference and stable endpoints are published, keep using LHCI for branch gates and the Watcher UI for schedules and alerts. Follow the Product & Brand category on the blog for product news and API-related updates, and check plan details for API access when you are ready to wire automation. If you are interested in early beta tester access to the agency API, contact us.

Checklist: ship a credible CI budget

[ ] Listed representative URLs (not only /).
[ ] Chose numeric thresholds aligned with your complete guide or client contract.
[ ] Set numberOfRuns ≥ 2 for stability.
[ ] Documented preview URL or static server startup in the workflow.
[ ] Added at least one resource or bundle guard for JS/CSS creep.
[ ] Separated error vs warn assertions to reduce alert fatigue.

FAQ

Is Lighthouse CI the only option?

No. Some teams wrap plain Lighthouse CLI, use Playwright traces, or rely on vendor-specific speed tools in CI. LHCI is widely documented and gives assertions + history-friendly uploads out of the box.

Should performance CI block every PR?

Often yes for main, with optional paths for docs-only changes. Use path filters so markdown edits do not run full LHCI unless you want them to.

Can I enforce budgets without a preview deploy?

Yes for static sites: build, serve locally in CI, and point LHCI at localhost. Dynamic server-rendered apps may need a test server with seed data.

Does this replace RUM or Search Console field data?

No. Lab CI validates the candidate build; field metrics validate real users. Both belong in a mature performance program.

What if our budgets are stricter than Lighthouse in CI can reliably hit?

Loosen CI thresholds slightly below production targets, or scope strict checks to stable audits (CLS, bundle size) and use warnings for high-variance metrics until your environment is stable.

Can I trigger Apogee Watcher tests from CI with an API?

That workflow is in active development. The customer API will support checking test results, triggering new tests, retrieving aggregated reports, and retrieving historical trends from automation for agency users on eligible plans (subject to published docs and plan limits). It is not a stable, copy-paste integration yet. For today’s deploy gates, use LHCI or Lighthouse in CI; keep Watcher for scheduled runs, budgets, and alerts. When the reference documentation and endpoints are public, they will be the supported way to align CI with dashboard budgets without putting a PageSpeed API key in your pipeline.

The Validation Bottleneck: Why Testing Is the New Speed Limit

Mon, 13 Apr 2026 13:00:00 +0000

AI coding agents can generate code in seconds. Your CI/CD pipeline cannot keep up.

That gap is the next major problem in software development. And it's one the industry is only beginning to take seriously.

For most engineering teams, the development feedback loop has three rough stages: write code, test it locally, push it through a CI/CD pipeline. The middle stage — local testing — has always been a limited proxy for what happens in production. Unit tests pass. Integration tests catch something else. Something breaks in staging that didn't show up anywhere else.

Agents don't change this dynamic. They make it worse.

More code, same pipeline

When a single developer using an AI agent can produce code at the pace that used to require a team, the volume of pull requests entering your pipeline goes up dramatically. The pipeline itself doesn't scale automatically. It's still spinning up test environments, running sequential jobs, waiting on staging queues.

"AI has solved everything but what mirrord does," said Aviram Hassan, CEO and co-founder of MetalBear. "You can generate code using AI. You can test it at a sort of local level — unit testing, component testing. You can review code. But once it comes to integration testing, you don't really have a good solution. And so that bottleneck just becomes more dominant, because there's more code that needs to be tested using those same limited resources."

Hassan's company, MetalBear, builds mirrord — a tool that lets local code run directly against a live Kubernetes cluster environment rather than a simulated local version. The premise is straightforward: local mocks don't reflect production conditions, and dedicated per-environment test setups are too slow and expensive to scale when agents are generating code at this pace.

Integration testing is the hard part

The layers of testing that AI assists with most naturally are the ones with the most deterministic feedback — unit tests and static analysis. Code either passes or it doesn't. An agent can read the error and try again.

Integration testing is different. It requires a full application context: connected databases, running services, real API dependencies. Setting up that context for each agentic code change — especially if you're running multiple agents in parallel — means either sharing a bottlenecked staging environment or spinning up dedicated environments that take time and cost money.

MetalBear's approach is to let agents access a shared production-like Kubernetes environment with traffic filtering and database isolation so multiple agents can test concurrently without stepping on each other. Early adopters have reported significantly faster CI cycle times compared to managing per-run ephemeral environments.

The broader point goes beyond any single tool. Traditional CI/CD pipelines were designed for a world where code moved at human speed. The architecture assumed a certain volume of commits, a certain frequency of deployments, a certain amount of time between changes. That assumption is breaking down.

What engineering teams need to rethink

The development pipeline needs to evolve in the same direction as the agents themselves — toward concurrency and faster feedback loops. A few practical shifts worth considering:

First, start measuring where your pipeline actually spends its time. In a lot of organizations, the majority of CI time is consumed by environment provisioning, not test execution. That's an infrastructure problem, not a testing problem, and it's often addressable without rearchitecting everything.

Second, think carefully about what local tests can and cannot tell you. An agent that passes unit tests is not necessarily producing code that works in your environment. The faster agents get, the more important it becomes to close that gap quickly rather than after a full pipeline run.

Third, parallel agent testing requires explicit isolation strategies. Shared staging environments were barely manageable when individual developers were competing for them. When multiple agents are running against the same environment simultaneously, you need deliberate approaches to traffic routing, database state, and queue handling.

The speed limit has moved

For most of the last decade, the speed limit on software delivery was writing code. Developers were the bottleneck. AI agents have shifted that constraint.

The new speed limit is validation. And unless testing infrastructure catches up, the productivity gains from agentic development will be partially absorbed by a pipeline that wasn't designed to handle them.

Deploying a Production-Ready React App on AWS using Terraform Module, S3, CloudFront & GitHub Actions

Wed, 08 Apr 2026 11:41:00 +0000

Modern frontend apps need fast, scalable, and cost-efficient hosting.
While AWS S3 is great for static hosting, making it production-ready
requires CloudFront, proper security, and automation.

In this blog, I’ll show how I built a fully automated pipeline.
to deploy a React app using Terraform and GitHub Actions.

Architecture Overview

Here’s what we’re building:

React app → built into static files
Terraform provisions:
- S3 bucket (hosting)
- CloudFront distribution (CDN)
GitHub Actions:
- Build React app
- Upload to S3
- Invalidate CloudFront cache

Project Structure

⚙️ Step 1: Terraform Module Design

Instead of writing everything in one file, I used modular Terraform.

Example modules:

How Modules Are Used (Root Configuration)

This snippet shows how the root module calls the S3 and CloudFront
modules to provision infrastructure.

module "s3_static_site" {
  source      = "./modules/s3-static-site"
  bucket_name = var.bucket_name
}

module "cloudfront" {
  source             = "./modules/cloudfront-distribution"
  bucket_domain_name = module.s3_static_site.bucket_domain_name
}

S3 Module (Static Hosting)

This module creates and configures the S3 bucket for static hosting.

Bucket Creation
This snippet creates the S3 bucket that will host the React static site.

resource "aws_s3_bucket" "skr_bucket" {
  bucket = var.bucket_name
}

Website Configuration
This snippet configures the S3 bucket for static website hosting and sets the index document.

resource "aws_s3_bucket_website_configuration" "skr_bucket" {
  bucket = aws_s3_bucket.skr_bucket.id

  index_document {
    suffix = "index.html"
  }
}

Bucket Policy (for CloudFront)
This snippet sets a bucket policy allowing CloudFront (and other services if needed) to read objects from the S3 bucket.

resource "aws_s3_bucket_policy" "allow_cloudfront" {
  bucket = aws_s3_bucket.skr_bucket.id

  policy = jsonencode({
    Statement = [{
      Effect = "Allow"
      Principal = "*"
      Action = "s3:GetObject"
      Resource = "${aws_s3_bucket.skr_bucket.arn}/*"
    }]
  })
}

CloudFront Module

This module sets up a CDN in front of S3.

Distribution
This snippet creates the CloudFront distribution that serves your S3 bucket globally with HTTPS support.

resource "aws_cloudfront_distribution" "cdn" {
  enabled             = true
  default_root_object = "index.html"
}

React Routing Fix (IMPORTANT)
This snippet ensures single-page React routes return index.html instead of 404 errors.

custom_error_response {
  error_code         = 404
  response_code      = 200
  response_page_path = "/index.html"
}

Cache Invalidation
This snippet invalidates the CloudFront cache automatically after deployment, so new changes appear immediately.

resource "null_resource" "cache" {
  provisioner "local-exec" {
    command = "aws cloudfront create-invalidation --distribution-id ${aws_cloudfront_distribution.cdn.id} --paths '/*'"
  }
}

This Terraform module-based architecture makes it easy to reuse infrastructure across multiple environments like dev, staging, and production.

Benefits:

Reusable
Clean architecture
Easier debugging
Production-ready structure

☁️ Step 2: S3 Static Website Hosting

The S3 module:

Creates a bucket
Enables static hosting
Configures public access (Via OAC)

Key features:

Versioning enabled
Proper bucket policy
Index + error documents

Step 3: CloudFront CDN Setup

CloudFront sits in front of S3 to:

Improve performance (low latency)
Add HTTPS support
Cache content globally

Configuration highlights:

Origin: S3 bucket
Viewer protocol: Redirect HTTP → HTTPS
Cache behavior optimized for static assets

⚡ Step 4: GitHub Actions CI/CD

This is where things get powerful.

Workflow does:

Install dependencies
Build React app
Sync build folder to S3
Invalidate CloudFront cache

Result:

Every push to main → automatic deployment

CI/CD Workflow Example

name: Deploy to CloudFront

on:
  push:
    branches:
      - main
  workflow_dispatch:

env:
  AWS_REGION: ap-south-1
  S3_BUCKET: samir-module-s3-bucket-hosting

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
          cache-dependency-path: static-site-react/package-lock.json

      - name: Install dependencies
        working-directory: ./static-site-react
        run: npm ci

      - name: Build React app
        working-directory: ./static-site-react
        run: npm run build

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Upload to S3
        run: |
          aws s3 sync ./static-site-react/dist/ s3://${{ env.S3_BUCKET }} --delete

      - name: Get CloudFront distribution ID
        id: cloudfront
        run: |
          DISTRIBUTION_ID=$(aws cloudfront list-distributions --query "DistributionList.Items[?Origins.Items[0].DomainName=='${{ env.S3_BUCKET }}.s3.${{ env.AWS_REGION }}.amazonaws.com'].Id" --output text)
          echo "distribution_id=$DISTRIBUTION_ID" >> $GITHUB_OUTPUT

      - name: Invalidate CloudFront cache
        run: |
          aws cloudfront create-invalidation --distribution-id ${{ steps.cloudfront.outputs.distribution_id }} --paths "/*"

      - name: Deploy success message
        run: |
          echo "✅ Deployment completed successfully!"

Why Use Terraform Modules?

Without modules:

Messy code
Hard to reuse
Difficult scaling

With modules:

Clean separation
Reusable across projects
Easier team collaboration

Key Advantages of This Setup

✅ Fully automated deployments
✅ Global CDN performance
✅ Infrastructure as Code (IaC)
✅ Scalable & production-ready
✅ Low cost (S3 + CloudFront)

How to Run This Project

Initialize Terraform

terraform init

Apply Infrastructure

terraform apply -auto-approve

Deploy React App

Push code to GitHub → CI/CD handles the rest

Best Practices

Use IAM roles instead of access keys
Store secrets in GitHub Secrets
Enable CloudFront caching strategies

Improvements You Can Add

Custom domain with Route 53
HTTPS with ACM
WAF for security
Multi-environment Terraform setup

Conclusion

This project demonstrates how to build a modern frontend deployment pipeline using:

Terraform Modules
AWS S3 + CloudFront
GitHub Actions

Final Output

After deployment, your app will be available via:

CloudFront URL (fast, secure, global)

Explore the Code

The full project with Terraform modules, S3, CloudFront setup, and GitHub Actions workflow is available here:
Check it out on GitHub

With this setup, you now have a fully automated, scalable, and production-ready React deployment pipeline. Start building modern frontends with confidence!Happy Building!!

Connect with me

• Portfolio: https://khanalsamir.com
• GitHub: https://github.com/Shawmeer
• LinkedIn: https://linkedin.com/in/samir-khanal

If you're working on similar DevOps or AWS projects, feel free to connect. I regularly share practical cloud and CI/CD content.

The Secret Behind SAST: The Security Blind Spot Developers Can’t Ignore (Part 1)

Fri, 16 Jan 2026 16:13:02 +0000

Itroduction

The first part of this article provides a robust theoretical foundation for understanding SAST and its importance in ensuring secure development. Additionally, the subsequent articles (Part 2, and Part 3) will delve into practical examples and effective implementation strategies.

In the field of modern software engineering, it is not only required to provide features on time but also to ensure their security. As companies refine their Secure Software Development Life Cycle (SSDLC) strategies, one essential aspect that stands out is Static Application Security Testing (SAST).

What is SAST?

SAST, or Static Application Security Testing, refers to a type of security testing that identifies and addresses potential vulnerabilities in software code. SAST involves examining source code, bytecode, or binaries without running the actual application by scanning the codebase for known vulnerability patterns.

Source code refers to the human-readable code written by developers.

Bytecode refers to the compiled form of source code that a virtual machine or runtime can execute, which can be analyzed by SAST tools to detect errors, vulnerabilities, or performance issues before deployment. For example, in Java .class files – the compiled Java classes generated from .java source files.

Binaries refer to compiled artifacts; in Java, these are typically .jar files, which are Java archives containing .class files and sometimes additional resources such as properties or XML files.

This tool can detect security weaknesses early on, which covers bugs, code smells, and the latest at this moment, 2021 OWASP TOP 10 categories, such as:

SAST tools analyze program components, including code structure, data flows, and control flows, to identify potential security vulnerabilities before the software is developed or deployed.

Control flow refers to the order in which individual statements, instructions, or functions in a program are executed.

Data flow refers to the movement of data through a program, specifically how values are assigned, modified, and passed between variables, functions, or modules. It defines how data travels and transforms during execution. One example is detecting tainted data. Tainted data refers to identifying untrusted or potentially dangerous input in a program that could flow into sensitive operations (called “sinks”) without proper validation or sanitization.

Control flow determines the sequence of execution in a program, while data flow tracks how data values propagate and transform through that execution.

Aspect	Control Flow	Data flow
What it describes	Execution order or decision paths	How data moves and changes
Examples	Loops, conditionals, function calls	Variable assignments, method parameters, and return values
Use in SAST	Detect unreachable code, infinite loops, and so on	Detect tainted data, SQL injection, and logging sensitive info

Some of the tools classified as SAST are:

Semgrep, PMD, SpotBugs, and FindSecBugs - popular open-source tools for code analysis and bug detection.
Fortify, Checkmarx, and Veracode - reliable enterprise solutions.
Snyk, SonarQube Cloud, and GitHub CodeQL - cloud-based or SaaS tools.

The importance of SAST in secure development cannot be overstated.

Implementing SAST within SSDLC enables the efficient implementation of large-scale security measures by:

Detecting problems early on
Halting the advancement of vulnerable code through the pipeline, SAST minimizes both time and expense associated with remediation.
Incorporating security earlier in the development process
Feedback is provided to developers during the commit or pull request process while the context is still fresh, allowing for any necessary fixes to be made more easily.
Promoting the education of developers.
SAST findings facilitate the organic acquisition of secure coding patterns by developers, promoting the integration of security as a daily practice.
Minimizing risk in the manufacturing process

By addressing vulnerabilities early on in the SDLC or live systems, there is a lower occurrence of security incidents. Several compliance frameworks, such as ISO 27001, SOC2, and PCI DSS, require the inclusion of static security testing in secure development governance.

How to integrate SAST into CI/CD?

The integration of SAST into the development lifecycle is essential for its optimal effectiveness.

Developer local environment (can be optional, but recommended)

Run lightweight SAST tools locally (Semgrep, CodeQL CLI, Snyk CLI, SonarQube local server using Docker) or run IDE plugins such as SonarQube Cloud that can connect easily from the IDE to the SonarQube Cloud server before pushing code.
Helps reduce noise in the CI pipeline or incidentally pushes sensitive data to the GIT repository.

Pull request (PR) and merge request (MR) stage

Protect the main branches from merging directly until all pipeline jobs are SUCCESSFUL. If Git Flow is used, protected branches typically are: dev, qa, or staging, pre-prod, and prod.
Trigger a SAST scan on each PR/MR.
Block merging if findings exceed severity thresholds, for example, by configuring Quality Gate in SonarQube or other severity-related policies. You can configure the CI/CD pipeline to FAIL if, for example, 1 Critical and/or 3 High vulnerabilities are detected.
Use the SAST tool’s suggestions on how to resolve the issue.
Investigate the findings to identify potential false positives. SAST scanners provide useful information about each vulnerability, making it easier to understand them.

CI pipeline

Add SAST as an early stage. Depends on the project; it can be after the build and test stage or before it, if only the source code is used to scan the project, but the recommended approach is to use it after, because you can configure it to scan for binaries or bytecode.
Fail builds based on:
severity level (Low, Medium, High, Critical)
the count of severity levels (for example, 20 Medium, 5 High, 2 Critical)
security policy rules
code smells

Scheduled full scans

Run nightly or weekly deep scans on the entire codebase. For example, Snyk scans the project by default once a week and sends the report to the user’s email (this is a SaaS tool, and it depends on whether the company is allowed to send the codebase to other services for scanning for vulnerabilities).
Capture findings not detected in incremental scans.

Governance

Export results into an issue tracker (such as Jira) or a SIEM (Security Information and Event Management) system. Some of the SAST tools allow the creation of an issue ticket in Jira.
Track remediation SLAs (Service Level Agreement) by severity:
Critical issues fixed within 24 hours
High-severity fix within 3 days
Medium 7–14 days
Low 30+ days
Maintain centralized dashboards (SonarQube, Snyk, DefectDojo).

The key principle: SAST must become a non-negotiable quality gate, just like unit tests, linters, or build verifiers.

Why do some companies run SAST within CVS instead of SaaS tools?

While modern SaaS security platforms are powerful, many enterprises still run SAST inside their version-control system (CVS) or self-hosted environments.

Data protection and compliance

Source code is often the company’s most sensitive intellectual property.
Some industries, such as finance, defense, and telecom, cannot upload code to external SaaS due to:

Data residency restrictions
Regulatory constraints
Confidentiality concerns

Full control and custom rules

Self-hosted SAST (SonarQube Enterprise, Checkmarx On-Premise) offers:

Custom rule tuning for domain-specific patterns
Control over scanner performance and scheduling
On-prem log retention and audit trails

No external vendor dependency
Large organizations want predictable SLAs and reliability without relying on external service uptime or rate limits. Companies often use cloud VMs (Virtual Machines) with autoscaling that make the server available even if one goes down, especially when all infrastructure is deployed on a specific cloud provider like AWS, Azure, and GCP. Still, it is more difficult to configure and maintain your own server compared to SaaS alternatives.

Cost optimization at scale

SaaS SAST tools are charged per seat or LOC (lines of code).
For large organizations with millions of LOC, on-premise licensing might be more efficient.
This doesn't make SaaS solutions inferior; they are excellent for small to mid-sized companies or teams that prefer zero maintenance overhead, but big enterprises often choose to keep scanning where the codebase is located or self-hosted.

How does SAST strengthen your application before deployment?

SAST adds many layers of security reinforcement before one line is deployed:

Eliminates vulnerabilities at the earliest point. Fixing issues in development prevents vulnerabilities from ever entering later stages.
Reduces attack surface. Every resolved injection flaw, every sanitized input, and every removed secret reduces the exploitable surface in production.
Improved code quality. Most of the SAST rules lead to improved code clarity, design patterns, and maintainability.
Drives security culture. SAST is an enabler of secure software development practices. Teams start thinking in terms of:
safe patterns
least-privilege design
secure defaults
flow validation

SAST best practices

To maximize the value of SAST in SSDLC:

Start with a minimal and focused rule set. Avoid overwhelming developers. Enable rules progressively.
Integrate into PR pipelines.
Make SAST feedback part of the daily development routine.
Use severity thresholds. These configurations can differ from company to company.
Avoid false positives. Not all findings are equal; that is why you need to tune rules to reduce false positives. Even if the finding’s severity level is High and it may appear to be a serious issue, it depends on how the feature or script was implemented.
Establish SLAs for remediation.
For example:
Critical: fix within 1-2 days.
High: fix within 5-7 days.
Medium: fix within 30 days.
Combine with other AST tools. SAST is one part of the security puzzle. Always complement it with:
ISCA (dependency scanning), if the tool does not support it, or you can use the free trial version of the tool.
WAF (Web Application Firewall) adds a defence layer in front of your application by filtering and monitoring HTTP/HTTPS traffic.

WAF does not replace SAST or DAST. It can block known attack patterns, mitigate exploitation attempts before fixes are deployed, provide real-time protection in production, and prevent DDoS attacks.

DAST (Dynamic Application Security Testing). Scan the DEV or QA environment, because DAST is used when the application is up and running.
Effective DAST can be performed in two phases:

Automated scanning — baseline detection of common runtime vulnerabilities.

Manual exploratory testing — for issues that automated tools often miss (logic flaws, authorization bypasses, etc.). Not all vulnerabilities can be discovered by SAST alone, which is why DAST is an important complement.

Ensure DevSecOps collaboration.
Security engineers, platform teams, and developers must collaborate on:
rule tuning
prioritization
education

Conclusion

SAST remains one of the cornerstones of a mature SSDLC. It provides early, actionable insights into code security, reduces risk, empowers developers, and fortifies applications before deployment.
Implemented with care, SAST becomes more than just a detection tool; it becomes a catalyst for a long-term secure engineering culture.

Closing

This article covered the theoretical foundation of SAST and its strategic role within secure development. In ”Catch vulnerabilities before they ship: local SonarQube setup (Part 2)”, we will explore SonarQube Cloud configuration and IntelliJ IDEA integration, as well as running SonarQube locally with Docker.
If you're building secure systems at scale, SAST is where prevention truly begins.

Hello World to GitHub Actions: Your First Automated Workflow

Thu, 25 Dec 2025 06:23:00 +0000

Hey there! So you're building a React app with TypeScript and Vite, and you keep hearing people talk about CI/CD and GitHub Actions. What's all the fuss about? Let me break it down for you in plain English.

What's GitHub Actions Anyway?

Think of GitHub Actions as your automation buddy that lives in your GitHub repo. You tell it "hey, whenever I push code, run my tests and make sure nothing broke," and it just... does it. Automatically. While you grab coffee(or chai) or work on something else.

Pretty cool, right? Here's what makes it awesome:

It's automatic - No more remembering to run tests before you push. Your automation buddy handles it.

It's consistent - Everyone's code gets tested the same way, every single time. No more "but it works on my machine!"

It's free - GitHub gives you a generous amount of free minutes every month. For most personal projects, you won't pay a dime.

It's built-in - Everything happens right in GitHub. No need to sign up for yet another service.

The Basics (Don't Worry, It's Simple!)

Before we jump into code, let's get familiar with a few terms:

Workflows are like recipes. They're files that tell GitHub Actions what to do. You write them in a language called YAML (it's just a fancy text format, nothing scary).

Events are triggers. "When someone pushes code" or "when someone opens a pull request" - these are events that kick off your workflow.

Jobs are the big tasks in your recipe. Like "build the app" or "run tests." Each job runs on its own fresh computer in the cloud.

Steps are the small tasks inside a job. Things like "install dependencies" or "run the linter."

Actions are pre-made shortcuts. Instead of writing commands to set up Node.js from scratch, you can use an action someone else already made. It's like using a package from npm, but for your workflows.

Let's Build Your First Workflow!

Alright, enough theory. Let's make something real. We're going to create a workflow that checks your code every time you push or open a pull request.

Creating the File

First, make a new folder in your project: .github/workflows/. Yes, it starts with a dot - that's on purpose!

Inside that folder, create a file called ci.yml. Here's what goes in it:

name: CI Pipeline

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main, develop ]

permissions:
  contents: read

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run linter
        run: npm run lint
      
      - name: Type check
        run: npm run type-check
      
      - name: Build project
        run: npm run build
      
      - name: Run tests
        run: npm test

What's All This Mean?

Let me walk you through it:

name: CI Pipeline - Just a friendly name for your workflow. You'll see it in the GitHub interface.

on: - This is where you say "run this when..." In our case, we're saying "run when someone pushes to main or develop, OR when someone opens a pull request."

permissions: - This is important! We're telling GitHub "hey, this workflow only needs permission to read our code, nothing else." It's like giving someone a library card instead of the keys to the building. Good security practice!

jobs: - Here's where the actual work happens. We've got one job called "build-and-test."

runs-on: ubuntu-latest - This means "run on a computer with Ubuntu Linux." GitHub provides this for free!

steps: - These are the actual commands that run, one after another:

First, we grab the code
Then we install Node.js
Install our dependencies
Run the linter to check code style
Check for TypeScript errors
Build the project
Run our tests

Setting Up Your package.json

Your package.json needs to have these scripts for the workflow to work:

{
  "scripts": {
    "dev": "vite",
    "build": "tsc && vite build",
    "lint": "eslint . --ext ts,tsx --report-unused-disable-directives --max-warnings 0",
    "type-check": "tsc --noEmit",
    "test": "vitest run",
    "preview": "vite preview"
  }
}

Don't have ESLint or Vitest set up yet? No worries - you can skip those steps in your workflow for now and add them later.

Want to Deploy Automatically Too?

Once your tests pass, you might want to deploy your app. Here's a workflow that deploys to GitHub Pages:

name: Deploy to GitHub Pages

on:
  push:
    branches: [ main ]

permissions:
  contents: read
  pages: write
  id-token: write

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Build project
        run: npm run build
      
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v3
        with:
          path: ./dist
  
  deploy:
    needs: build
    runs-on: ubuntu-latest
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    
    steps:
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v4

This one's a bit different - it has TWO jobs. The second job (deploy) waits for the first one (build) to finish successfully. That needs: build line is what makes that happen.

Making It Faster with Caching

Want your workflows to run faster? Good news - the actions/setup-node@v4 action we're using already caches your npm packages when you add cache: 'npm'. That means the second time your workflow runs, it won't have to download all your dependencies again. Pretty smart!

Keeping Secrets... Secret

Sometimes you need API keys or passwords in your builds. DON'T put them directly in your workflow file - that's like posting your house keys on Instagram.

Instead, use GitHub's secrets feature:

Go to your repo's Settings
Click on "Secrets and variables" then "Actions"
Click "New repository secret"
Add your secret (like VITE_API_KEY)

Then use it in your workflow like this:

- name: Build project
  run: npm run build
  env:
    VITE_API_KEY: ${{ secrets.VITE_API_KEY }}

GitHub will hide it in the logs automatically. Nice!

Checking If It's Working

After you push your workflow file to GitHub:

Click the "Actions" tab in your repo
You'll see your workflow running (or done running)
Click on it to see all the details
If something breaks, click on the red X to see what went wrong

The logs tell you EXACTLY what happened at each step. It's super helpful for debugging.

Understanding Workflow Permissions

Quick important thing - see that permissions block we added? That's like setting up proper door locks for your house.

By default, workflows might have way more permissions than they need. We're being explicit and saying "this workflow only needs to READ the code" with contents: read. If later you need to do something like comment on pull requests, you'd add pull-requests: write.

It's just a good habit to be clear about what your workflow can and can't do. Your future self (and your team) will thank you!

Some Quick Tips

Here are a few things that'll make your life easier:

Set permissions explicitly - Like we just talked about, always add that permissions block. Start with contents: read and only add more when you need it.

Use version tags - Notice how we use @v4 in our actions? That's better than using @main because it won't suddenly change on you. Your workflows stay stable.

Use npm ci not npm install - In automated environments, npm ci is faster and more reliable. It's designed specifically for CI/CD.

Run jobs in parallel - If you have multiple jobs that don't depend on each other, they'll run at the same time. Saves time!

Add a badge to your README - In the Actions tab, you can grab a badge that shows if your build is passing. It looks professional!

Test different Node versions - You can use something called a matrix to test on Node 18, 20, and 21 all at once. Here's how:

strategy:
  matrix:
    node-version: [18, 20, 21]

Protect your main branch - In your repo settings, you can require that all checks pass before merging. No more broken builds!

Separate your workflows - Keep your testing workflow separate from your deployment workflow. Makes things easier to understand and manage.

When Things Go Wrong

Don't panic! Here are common issues and fixes:

Workflow not running? - Check that your YAML file is in .github/workflows/ and that you don't have any syntax errors. YAML is picky about spaces and indentation.

Works on your computer but not in CI? - Make sure all your dependencies are listed in package.json. Also check that you're not relying on environment variables that only exist on your machine.

Running out of build minutes? - Public repos get unlimited minutes! Private repos have limits. Check your usage in Settings → Billing.

What's Next?

You now know the basics! Here are some cool things to explore when you're ready:

Browse the GitHub Marketplace for actions other people have made
Set up notifications to Slack or Discord when builds fail
Add code coverage reporting
Create custom actions for your specific needs
Try matrix builds to test across multiple environments

The key is to start simple - get your basic testing pipeline working - then add more features as you need them. Don't try to do everything at once!

Wrapping Up

GitHub Actions might seem a bit intimidating at first, but once you get the hang of it, you'll wonder how you lived without it. No more forgetting to run tests. No more "oops, I broke main." Just push your code and let your automation buddy do its thing.

Start with the simple CI workflow we built together, make sure it works, and then gradually add more automation as you get comfortable. Before you know it, you'll be the GitHub Actions expert on your team!

Happy automating!

The difference between deployments and releases

Mon, 08 Dec 2025 15:20:22 +0000

In software delivery, it’s common to use the terms release and deployment to refer to the same thing: making software available to users. The terms are traditionally synonyms because they happen at the same time. However, modern software delivery practices encourage us to separate the concepts.

Let’s define the terms with more precise language and look at the practical benefits of the distinction:

Deployment is when you install a software version on an environment
Release is when you make software available to a user
You could use the word release to describe a package ready for use outside the development team. However, by referring to this as a software version, you can avoid getting into a situation where you release (verb) a release (noun). Instead, you deploy or release a software version.

Technical and business tension

When DevOps was created, it recognized a natural tension between development and operations. The development team wanted to deploy more often to get new features out, while the operations team wanted to deploy less often to keep the system stable. The goal of DevOps was to resolve this tension.

If your releases are linked to deployments, you’ll find a similar tension between development and the business. You might have had a situation where the development team wanted to deploy a software version, but the business wanted to hold back the launch for marketing activities or a trade show.

By decoupling releases from deployments, developers can deploy to production frequently and the business can decide when to launch specific features.

A timeline for change

There’s a 4 step timeline for decoupling releases from deployments in your organization:

Adopt the precise terms
Introduce the architecture
Start deploying and releasing independently
Transfer control

Let’s look at what those steps mean.

Adopt the terms

The language change is the easiest place to start. By increasing your precision when talking about deployments and releases, you can start thinking differently about the two different actions.

You also need to bring your business stakeholders along on this linguistic journey. Having a shared language between the business and technical team members reduces the amount of explanation needed when talking about the software and its deployment pipeline.

While you coach your organization in semantics, you can start the next step.

Introduce the architecture

There’s a high risk of introducing release controls that become hard to manage. Without a careful design, you could end up with conditional statements scattered through your system or with several competing release control toolsets.

That’s why your release control toolset is an architectural decision.

Most release control is done with feature toggles or feature flags.

A feature toggle is a simple on or off switch for a feature
A feature flag allows selection between different versions of a feature

You can reduce complexity by limiting how many feature controls you have and the states each control has. You should remove release controls by default once the feature’s available to all users. You can identify these regularly and remove them in the next deployment.

In some cases, you may decide to implement a long-lived release control. For example, a toggle that lets you temporarily remove a resource-heavy feature during a production incident.

Release controls can apply to all users or groups of users. You should limit the number of groups and the release controls you apply to these groups at one time. You can use group-based release controls for a canary release (where you test out a feature with a small group of early-access users) or for A/B testing two different feature implementations.

There’s helpful guidance on limiting complexity in this article on feature toggles by Pete Hodgson on Martin Fowler’s website.

Your design for release controls should include a level of audit logging. Audit logging has high value when you respond to bug reports or production incidents.

When you test your software, you should use realistic business configurations rather than trying to test every combination of release controls.

Release independently

With a release control in place, you can deploy more often with the feature switched off. You can then wait for the business to line up their launch activities and release the feature when they’re ready without needing a deployment.

You should find the introduction of the release control has resolved the tension between development and the business. However, the business still depends on an engineer to throw the switch. You can take this a step further in the final step.

Transfer control

You might decide to keep hold of the release controls while you gain confidence that everything works smoothly. In the longer term, you should make the appropriate controls available to authorized business users.

When doing this, ensure they only have release controls intended for business use. You don’t want to trigger a critical incident only to find someone accidentally switched off one of your resource-heavy features.

You should already have an audit log of release control changes. The audit log is even more vital as you extend the group of users who can make changes.

Summary

You can resolve classic developer and business tensions by separating releases and deployments. You can create a release control toolset that makes it possible to release features at runtime. You must design release control to limit the complexity and prevent conditional statements from appearing all over your code.

By gradually introducing the concept of independent releases, you can take the business people with you on the journey. They should be able to launch a feature at the push of a button, just like the development team should be able to deploy on demand.

Implementing an End-to-End CI/CD Pipeline for a Java ATM Simulation Application

Tue, 02 Dec 2025 23:48:32 +0000

Abstract
This case study outlines the design and implementation of a complete CI/CD pipeline for a Java ATM Simulation application. The project aimed to strengthen hands-on skills in DevOps automation, continuous integration, code quality analysis, artifact management, and deployment workflows using real-world tools on AWS.

Project Goals

Develop a fully automated CI/CD pipeline from code commit to deployment.

Integrate Jenkins, Maven, SonarQube, Nexus, and Tomcat into a streamlined workflow.

Deploy services on separate AWS EC2 instances to mimic production infrastructure.

Implement artifact versioning and reliable storage.

Enable automated deployment to an application server with minimal manual intervention.

Tools & Technology Stack

AWS EC2: Dedicated servers for Jenkins, SonarQube, Nexus, and Tomcat.

Jenkins: CI server and pipeline orchestration.

Maven: Build automation and dependency management.

SonarQube: Static code analysis and code quality reporting.

Nexus Repository Manager: Artifact storage and version control.

Apache Tomcat: Application server deployment.

GitHub: Source code management.

Java 11+: Application and tooling requirement.

Jenkins Plugins: Git, Maven, SSH Agent, SonarQube Scanner.

Infrastructure Setup
Four EC2 instances were provisioned, each hosting a separate tool to simulate production:

Jenkins on port 8080

SonarQube on port 9000

Nexus on port 8081

Tomcat on port 8080

Java was installed on all instances, and security groups were configured for inter-tool communication. Each server was validated for accessibility via a browser to ensure proper connectivity.

Pipeline Design & Implementation

Source Control Integration
The ATM Simulation project was pushed to GitHub. Jenkins was configured to clone the repository automatically on each build trigger using a declarative pipeline script.

Build & Test Automation
mvn clean package was used to compile code, execute tests, and generate artifacts. Maven ensured dependency management and consistent builds across environments.

Code Quality Analysis
SonarQube was integrated by updating pom.xml and Jenkins credentials. Running mvn sonar:sonar generated detailed code metrics, highlighting vulnerabilities and enforcing coding standards.

Artifact Management
A Maven repository was created in Nexus. pom.xml and Jenkins settings.xml were configured to upload artifacts via mvn deploy, ensuring version-controlled storage and rollback capability.

Automated Deployment to Tomcat
Tomcat manager roles and credentials were configured. Jenkins deployed the final WAR file, completing the continuous delivery process. Poll SCM triggers enabled automation for every commit.

Advanced Jenkins Configuration
The SSH Agent plugin simulated a master-slave architecture, enabling distributed builds and enhancing pipeline scalability.

Challenges & Solutions

Tool Version Conflicts: Resolved Java compatibility issues for SonarQube and Nexus.

Nexus Deployment Failures: Fixed misconfigurations in pom.xml and settings.xml.

Tomcat Deployment Errors: Corrected manager roles and permissions for successful deployment.

Pipeline Failures: Updated Jenkins plugins and validated pipeline scripts to resolve build issues.

Key Learnings

Hands-on experience in end-to-end CI/CD workflows.

Integration between source control, CI servers, artifact repositories, and application servers.

Best practices in artifact versioning, storage, and automated deployment.

Troubleshooting pipeline failures and ensuring tool compatibility.

Leveraging Jenkins plugins for scalable master-slave architectures.

Real-world automation of software delivery and cloud deployment.

Conclusion
This project provided practical exposure to enterprise-grade DevOps practices, including automation, CI/CD pipeline creation, artifact management, and deployment orchestration. It strengthened problem-solving, tool integration, and cloud deployment skills, demonstrating the ability to implement and manage robust DevOps workflows in real-world environments while reinforcing a commitment to continuous learning in automation and CI/CD.

Deploy your application on Vercel with GitHub Actions

Wed, 20 Aug 2025 17:26:29 +0000

A very common CI/CD pattern for frontend apps is: build in CI and deploy to production automatically when the code reaches main.
If you use Vercel, you can do this with a short and reliable GitHub Actions workflow that:

downloads your project's configuration and variables from Vercel,
builds the app in CI,
and deploys the already built artifacts to production.

In this article, we'll walk through it step by step and I'll give you some tips to help you understand what each line does.

Purpose of the workflow

Every time there is a push to main, the following should be executed:

Pull config from Vercel (includes production environment variables).
Reproducible build in CI.
Deploy that artifact (without rebuilding in Vercel).

This will give you predictable deployments that are easy to audit and debug.

General workflow structure

name: Vercel PRO deployment
on:
  push:
    branches:
      - main
jobs:
  deploy:
    name: deploy
    runs-on: ubuntu-latest
    steps:
      - id: checkout-step
        name: Checkout code
        uses: actions/checkout@v4
      - id: install-vercel-cli-step
        name: Install Vercel CLI
        run: npm install --global vercel@latest
      - id: pull-vercel-env-info-step
        name: Pull Vercel Environment Information
        run: vercel pull --yes --environment=production --token=${{ secrets.VERCEL_TOKEN
          }}
      - id: build-vercel-artifacts-step
        name: Build Project Artifacts
        run: vercel build --prod --token=${{ secrets.VERCEL_TOKEN }}
      - id: deploy-vercel-artifacts-step
        name: Deploy Project Artifacts to Vercel
        run: vercel deploy --prebuilt --prod --token=${{ secrets.VERCEL_TOKEN }}

Prerequisites
Create the secret VERCEL_TOKEN in your repository (Settings → Secrets and variables → Actions).
Either commit .vercel/project.json, or provide VERCEL_ORG_ID and VERCEL_PROJECT_ID as secrets when you do vercel pull.

Step by step

on: push (to main)
Each push to main triggers the production pipeline. If you prefer to only deploy when merging by PR, simply avoid pushing directly to main.

uses: actions/checkout@v4
Download your code to the runner so that the following tools (Vercel CLI) have it available.

npm install --global vercel@latest
Install the Vercel CLI.

Tip: to avoid surprises, you can set version: vercel@xx (or whatever you use on your team).

vercel pull --yes --environment=production
Bring the project configuration and production environment variables to .vercel/ on the runner.
In CI there are no interactive questions, so we use --yes.

You need:

.vercel/project.json in the repository or
VERCEL_ORG_ID and VERCEL_PROJECT_ID as variables/secrets.

vercel build --prod
Builds in CI using your framework (Next.js, SvelteKit, etc.) or what is defined in vercel.json.
The result is stored in .vercel/output and is exactly what you are going to deploy.

vercel deploy --prebuilt --prod
Uploads the already built artifact to production.
With --prebuilt, Vercel does not rebuild: what you tested in CI is what your user will see.

✅ Final result

With this workflow, each merge to main runs a clean build and deploys that same artifact to production in Vercel. You get:

reproducible builds,
faster deploys,
and clear CI logs for debugging.

Prefer to configure it visually?

Writing workflows by hand gives you total control, but sometimes you want speed with built-in validations.

With OctoLab, you can set up this same pipeline visually: choose actions, fill in fields, and the YAML is generated instantly, ready to commit.

Try it out: https://www.octolab.app/templates/vercel-pro-deployment

Conclusion

This workflow provides you with a clean CI → production flow on every push to main, using Vercel's recommended approach: pull → build → deploy. It's easy to adapt to previews, monorepos, or strict reproducibility (fixed Node and CLI).

Use it as a base and evolve it with caching, notifications, or environment gates as your team grows.

Publish your packages to NPM automatically with GitHub Actions

Tue, 19 Aug 2025 11:21:15 +0000

Automating publication prevents manual errors, speeds up releases, and forces you to maintain a repeatable and transparent process.

In this article, we're going to create a GitHub Actions workflow that publishes to NPM when you push to main. The flow installs dependencies, runs tests, compiles, and, if all goes well, publishes the package using a secure token stored as a secret.

What is the purpose of the workflow?

Every time you merge changes to main, we want to:

Install dependencies
Run tests
Build the package
Publish to NPM using secure credentials

This standardizes the publishing process and reduces the risk of human error.

General workflow structure

name: Publish to NPM
on:
  push:
    branches:
      - main

jobs:
  publish:
    name: publish
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: "22"

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm run test

      - name: Build package
        run: npm run build

      - name: Publish to NPM
        uses: JS-DevTools/npm-publish@v3
        with:
          token: ${{ secrets.NPM_TOKEN }}

️ ️ What does each step do?

on: push to main

Triggers the workflow when pushing to the main branch. If you prefer to publish only with tags, you can change the trigger to on: push: tags: - ‘v*’.

actions/checkout@v4

Downloads the repository to the runner.

actions/setup-node@v4

Configures Node.js (v22 in the example). You can enable cache: ‘npm’ to speed up installations.

npm ci / npm install

Installs dependencies. npm ci is faster and more reproducible in CI if you use package-lock.json.

npm run test and npm run build

Verify that the package passes tests and compiles correctly before publishing.

JS-DevTools/npm-publish@v3

Publish the package by reading package.json. Use the token you pass through with.token.

Authentication with NPM: NPM_TOKEN

To publish, you need a token in NPM and save it as a secret in GitHub:

Create an Automation Token in your NPM account (recommended for CI; respects 2FA and allows automatic publishing).
In GitHub, go to Settings → Secrets and variables → Actions → New repository secret.
Create the secret NPM_TOKEN with the value of the NPM token.

If your package is scoped (e.g., @your-org/package), make sure that package.json contains “name”: “@your-org/package” and “publishConfig”: { ‘access’: “public” } if you want it to be public.

Example of publishConfig in package.json:

{
  "name": "@tu-org/tu-paquete",
  "version": "1.2.3",
  "publishConfig": {
    "access": "public",
    "tag": "latest"
  }
}

Dry-run and version control

Dry-run: test the flow without actually publishing:

- name: Publish to NPM (dry run)
  uses: JS-DevTools/npm-publish@v3
  with:
    token: ${{ secrets.NPM_TOKEN }}
    dry-run: true

Versioning: this flow publishes the version specified in package.json. Be sure to update the version before merging to main.
If you use automatic versioning (e.g., Conventional Commits + automatic releases), integrate that stage before the publication step.

✅ Final result

With this workflow:

You publish to NPM in a consistent and repeatable manner.
You avoid broken releases thanks to preliminary tests + builds.
You keep credentials secure with GitHub Secrets.
You reduce manual steps and time between merge and release.

Would you rather set it up with a visual interface?

If you're interested in defining this flow with a visual interface and getting the YAML instantly, you can use OctoLab:

Step-by-step visual editor
Dynamic fields for actions and tokens
Real-time YAML preview
Built-in validations
Copy/download and you're done

Try it out: https://www.octolab.app/templates/npm-publish

Conclusion

Automating publication to NPM with GitHub Actions reduces friction and errors. With a simple configuration—tests, build, and a publish step with a secure token—you can standardize releases and focus on building value.

If this was helpful, let me know what you would like to add to the template (tags, prereleases, monorepos, changelogs, etc.). Let's keep going!

Beyond skeleton pipelines: who owns your software pipeline?

Thu, 14 Aug 2025 00:53:24 +0000

Your current software delivery processes are probably working fine. They build your code, maybe run a test or two, and get your software to production. That's no small achievement, but here's a question that might make you pause.

When did someone last spend time improving them?

If you're like many software teams I've worked with, the answer is probably "not recently". Maybe not ever.

I've worked with developer teams at different maturity levels and have repeatedly seen this pattern. Great development teams who are busy building software but aren't large enough to have platform teams or dedicated operational folk responsible for the software delivery process. They typically start with a template from their cloud provider, delivery tooling vendor, or, these days, their favorite AI LLM. Initially, it gets the job done, but I refer to these as skeleton pipelines—the bare minimum framework to ship code.

Skeleton pipelines are a great starting point. They get you from nothing to shipping code quickly, but your pipelines are living, breathing artifacts. Before long, they need some meat added to the skeleton to provide functionality beyond the basics and make your delivery process robust and delightful.

I've seen when teams experience friction with their software release process, but don't treat it as a bug or feature improvement that needs to be logged, prioritized, and worked on. If your deployment pipelines are responsible for delivering your product to customers, shouldn't you treat them as part of your product? When was the last time someone added a pipeline improvement to your backlog or spent time making your deployment process better?

The important shift is treating delivery process improvements as regular development work, not spare-time projects that inevitably get pushed aside.

The ownership vacuum

Ask any group of engineers, "Who owns your software delivery pipelines?" and I bet you'll get different answers. When writing this post, I put that question into a search engine and reviewed the discussions, which weren't surprising. Some say developers should own what they build, others insist it's up to platform or DevOps engineers. Then there's the view that everyone is responsible because DevOps is about everyone working together. My observation is that often, no one owns it, and that's when it becomes a problem.

The distinction that matters isn't really about ownership. It's about accountability. Someone needs to be accountable for ensuring your delivery process evolves, improves, and supports your organization with shipping software rather than being a friction-filled process people loathe working with. Without that accountability, pipelines can easily become unmanaged components that gradually deteriorate until they become technical debt and a bottleneck instead of an enabler.

There's no one-size-fits-all answer for who should own and be accountable for your pipelines. You must discuss and agree based on your maturity, size, team structure, and individual skill sets. The important thing is that it is discussed and agreed upon, and not left to assumptions. Ask yourself now if everyone on the team knows how and where to raise an issue with the deployment process today. If the answer is no, then start there. Determine who's responsible and establish clear channels for pipeline feedback, problems, and improvements.

The cost of skeleton pipelines

This ownership vacuum creates predictable problems I've seen many times. Teams fall into the "if it ain't broke, don't fix it" mentality, leaving pipelines as-is for months or years. The issue is that what worked well 6, 12, or 18 months ago likely isn't fit for purpose today. New tooling and techniques exist now that didn't then. Your team will likely ship more frequently—or at least want to—and handle more complex deployments than when you first created that pipeline. The infrastructure you're deploying to has likely changed. Different developers and engineers might work at your company who didn't before. You're missing out on tooling and practices that could:

Save weekly hours
Bring new functionality to your software delivery lifecycle that adds business value
Catch issues before they become customer problems

Worse, they often become single points of knowledge. Whoever initially sets up the pipeline is the only person who understands how it works. When that person leaves or gets pulled onto other projects, your team inherits a mysterious system everyone fears touching.

Fragile pipelines break at the worst possible moments. They miss opportunities to catch issues early, optimize build times, have predictable outcomes, or provide better feedback to developers. Maybe, most importantly, they frustrate your team and slow down what they're supposed to enable: shipping great software and products.

You can't afford to ignore this

"We're too busy building features to worry about pipeline improvements." Sound familiar? It's a common pushback I hear from teams, and I get it. When trying to ship products and keep customers happy, spending time on what feels like an internal process is often considered a luxury you can't afford.

Platform teams are getting a lot of attention lately, and for good reason, as they work well for larger organizations. But if your company doesn't need everything else a platform team typically handles, creating one just for pipeline ownership is unnecessary. The reality is that you don't need a dedicated platform team to own your pipelines. You can borrow their mindset by treating your delivery process as part of your product. Pipeline improvements get backlog items, performance issues get bug tickets, and deployment pain points get the same attention as user-reported problems.

Start by paying attention to where your process hurts. If deployments make you nervous, your team loathes "release day", when releases fail for mysterious reasons or when you're manually testing things your team can automate. These pain points are your roadmap for high-value improvements.

Making the investment

Improvements may include:

Adding vulnerability scanning during builds
Configuring integration tests against ephemeral infrastructure
Properly storing and versioning your deployment artifacts
Implementing flexible approval workflows
Creating a robust deployment strategy
Measuring deployment metrics

I'm not giving you a shopping list of things to do. Instead, I recognize that "you don't know what you don't know" because I've been there too, and you may be experiencing issues you didn't know there were solutions to.

We have a white paper titled The ten pillars of pragmatic deployments. I bring this to your attention not for a checklist of everything you should do but for insight into what we think good deployments look like based on years of experience building a deployment automation tool. The ideas discussed in this post might help you enhance your software delivery process today or provide insight into issues and solutions you may not have been aware of.

Take ownership

It's easy to overlook software delivery pipelines. They're usually out of sight until they get in your way, kind of like plumbing. It's invisible when it works and catastrophic when it doesn't. Excellent plumbing supports everything in your house to function smoothly, and great delivery processes do the same for your product development.

Claim ownership of your software delivery processes and ensure someone is accountable for their care and feeding. Talk to your engineers involved in developing and releasing software, and find out where the pain is. Implement easy-to-use feedback loops, add pipeline improvements to your backlog, and treat deployment pain points as bugs worth investing in and fixing.

Your future self and team will thank you when your next deployment goes smoothly instead of keeping everyone up after hours or nervously huddling around a few pizzas doing your monthly release with crossed fingers.

Go Ahead! Deploy on Friday!

Mon, 11 Aug 2025 13:12:30 +0000

Improving your deployment pipeline reduces the risk of failed deployments overall rather than choosing which day will be a bad day.

Avoiding Friday deployments is driven by a goal that, while admirable, is better achieved in other ways. Yet, organizations are more likely to deploy on Friday than on a Monday, which shows the industry doesn’t buy the myth that you shouldn’t deploy on Friday.

Few Practices Are Truly Best Practices

While you used to hear about best practices all the time, you’ll have noticed that industry experts have become increasingly cautious about the term. Best practices do exist, such as using version control instead of a network share to store your source code. Many other practices turn out to be simply good, rather than best.

The idea of best practices is that they are table stakes for software delivery, and working without them can be harmful. Good practices, on the other hand, are options. You don’t have to adopt every good practice, as you can cover risk areas with a carefully selected option from the menu of all tried-and-tested ways to solve a problem.

If your end users are reporting escaped bugs, you can apply one or more testing and monitoring good practices to catch similar issues before users experience them. You look at different testing and monitoring approaches and try one to see its effectiveness. The problem could be solved after you adopt one practice, or it might take several complementary approaches to improve the situation.

Crucially, you should stop adding practices once you’ve solved the problem. Adding further practices that aren’t required results in unnecessary complexity, which can be harmful to software delivery.

Finally, there are practices that are actively harmful when pursued. These are often well-intentioned and almost always presented as best practices, but empirical evidence and rigorous research have proven them to be false. Examples of these harmful practices include Gitflow, working in large batches, and heavyweight change-approval processes.

A good heuristic for spotting an anti-pattern is that you’ve being told something is a best practice, but it doesn’t sound like table-stakes basic software delivery. If there’s any doubt, look for trusted advice from someone with plenty of experience and cross-reference dependable research, such as the Accelerate State of DevOps Report.

Never Deploy on a Friday

With the practices primer firmly in mind, let’s consider whether we should deploy on Friday. After the recent CrowdStrike outage, many people suggested things could have been improved if they hadn’t deployed on Friday.

It’s hard to understand how deploying on Thursday would have improved the CrowdStrike issue. Do we not fly planes or use computers on Thursday? We can explore this confusing claim by looking at the perceived benefits of banning Friday deployments.

Why You Should Avoid Friday Deployments

I asked folks why we should avoid Friday deployments to see if there was some merit to the claims. As someone whose mind was changed on tabs vs. spaces by solid arguments, I believed I could be convinced given a compelling argument.

The most common argument for a Friday moratorium was that it would allow developers to spend the weekend with their families instead of working to restore services.

This is only true if your time to recover from a failed deployment is almost exactly a day. If it takes more than a day, you’ll be working on Saturday to recover from your Thursday deployment, unless you also stop deployments on Thursday. If you can recover in less than a day, you could deploy on Friday and still get to the beach on Saturday.

The longer your recovery times, the further in the past you need to deploy to avoid working on the weekend. For example, if it takes you a week to recover from a failed deployment, you need to deploy last Friday to avoid working this weekend, though this does mean you have to work last weekend.

The reality is that recovery times aren’t constant. Some problems are easier to recover from than others. Having a distribution of recovery times means you are taking a probabilistic approach to protecting the weekend. If you deploy every day, you’ll discover that Friday deployments have a higher probability of disrupting the weekend than Monday deployments.

When you take this approach, you’ll find there are plenty of improvements you can make to your deployment pipeline that reduce the probability of disruption. Banning deployments doesn’t reduce your overall change failure rate, it just means your Monday deployments will fail more often. Improving your deployment pipeline reduces the risk of failed deployments overall, which is far more positive than simply choosing which day will be a bad day.

Work-life balance applies to Monday night just as much as it applies to the weekend. People have personal commitments every day of the week, from school runs to birthdays and anniversaries to arrangements made to meet an old friend. Each of these is important, so making improvements that reduce failed deployments protects work-life balance more than shifting them to a different workday.

Real Reasons To Plan Deployments

There are reasons to avoid deploying at specific times, but they are related to your industry. For example, in the retail industry, you avoid updating the software on tills when the store is open. You don’t need a new version of software while you’re checking out a customer and have a queue of shoppers waiting.

Your release plan would set out the times to deploy and the times to avoid. It would also capture the process for training retail store staff so they know what’s changed before they raise the shutters.

By thinking of the people affected by our software, we consider many more people than if we just think of ourselves. If we think of all the people depending on our software, we can consider the work-life balance of a thousand times more people. Ideally, we’d never have a failed deployment, and our rollout would have no downtime. In reality, things do sometimes go wrong, and having a strong recovery time can minimize the impact.

The research suggests there are capabilities that let you deploy more often while also reducing the failure rate. The top performers are deploying many times a day, including Fridays.

Deploying on Friday Is Commonplace

To understand whether the Friday deployment ban was affecting real organizations, I analyzed over 32 million deployments to see when they were taking place. The data cuts across all organization sizes and many industries. It turns out that deploying on Friday is more common than deploying on Monday.

This is great news, as it demonstrates that the myth of avoiding Fridays has little traction in practice.

Decide for Yourself but Don’t Propagate the Myth

It’s perfectly acceptable for your organization to decide they don’t want to deploy on Fridays, though some curiosity about why will either turn up interesting domain knowledge or highlight some things you can improve in your deployment pipeline. What’s less appealing is broadcasting “don’t deploy on Fridays” as a best practice.

Working in small batches is known to improve software delivery performance. If you are following continuous delivery and DevOps, you might be deploying five times a day. If you rule out Fridays, you build up a Monday deployment that’s five times bigger than your normal batch size. You are opting to downgrade your performance from “on demand” to “daily.”

The more continuously you deploy, the worse the idea of stopping for a day becomes. That means that the members the “don’t deploy on Friday” movement, in the absence of specific organizational context, are opting for mediocrity.

Don’t just start deploying on Fridays. Take this as an opportunity to assess your situation, understand users and customers, and start improving your software delivery capabilities.

CI/CD Uses

Tue, 05 Aug 2025 09:37:43 +0000

As I enter the new era of development. CI/CD is a must-have for modern engineers. It improves code quality and saves time. Starting a simple, automate things, and expand.

What is CI/CD?

CI/CD stands for

CI (Continuous Integration): Automatically test the code when changes are deployed.
CD (Continuous Development/Delivery): Automatically build and deploy the app for testing.

Tools can be use

GitHub - hosting repo and run Actions
VS code - to write code and create workflow files.
Git - to track changes and push to GitHub.
Vercel - This is already integrating with Git repositories.

Step-by-Step

My step-by-step guide when I am making a new repository and personal projects.

Create a GitHub repository
- Click new -> Name of the repo -> Add README.md
- Clone the repository:
```
  git clone https://github.com/username/repository.git
  cd repository
```

Setup the project

I can use any project - for example, a PHP or node.js app. Here's an example using PHP.

This is a manual setup

Inside my repo, I create:
```
touch index.php composer.json
```
I add a simple PHP file and install dependencies if needed.

This is a setup from GitHub Actions

Inside the repository, click the Actions tab
Search workflow. For example: PHP
Tap / Click Configure
Wait for the workflow finish the deployment.
Once the workflow deployed. Pull the updates from your local. Run the terminal and type:
```
    composer init
```
It will add new compose description. Fill the questions.

Create the GitHub Action Workflow

For Manual set up post. Inside the project, create the directory and file:

 mkdir -p .github/workflows
 touch .github/workflows/ci.yml

name: PHP CI

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up PHP
        uses: shivammathur/setup-php@v2
        with:
          php-version: '8.1'

      - name: Validate composer.json and composer.lock
        run: composer validate

      - name: Run tests (if any)
        run: phpunit

You can modify this workflow for JS, Python, etc.

Commit and Push to GitHub

In VS Code Terminal

 git add.
 git commit -m "Description here"
 git push origin main

Once the changes deployed to the repository, go to your GitHub Repo -> Actions Tab check the workflow running.

Optional: Add a simple test.

If your app has tests. GitHub actions will run automatically. Just make sure your app builds successfully.

Final Notice

If you successfully work with the CI/CD Actions. The next steps you can try is

Add testing
Set up deployment (CD)
Use secrets for environment variables.