Coder Legion - Recent posts tagged gitops

The GitOps Bridge: Keep Terraform and ArgoCD Out of Each Other's Way

Mon, 12 Oct 2026 04:24:41 +0000

🌉 The GitOps Bridge: Keep Terraform and ArgoCD Out of Each Other's Way

TL;DR

Infrastructure (Terraform) and applications (ArgoCD/Flux) have different change frequencies, approval flows, and blast radii. Coupling them is the root problem.

The GitOps Bridge is a narrow handoff: Terraform writes cluster facts into a well-defined interface (ConfigMap/Secret labels and annotations), and GitOps controllers read that interface to deploy.

Neither side imports the other's state. The interface is the only shared surface.

On a real migration this let three environments come up reproducibly, with the customer executing the IaC themselves under a zero-access constraint, and cut the timeline roughly in half.

Two tools, two clocks. Terraform provisions the cluster, the node groups, the networking, the IAM. It changes rarely and under heavy approval. ArgoCD deploys the applications onto that cluster. It changes many times a day under light approval. The trouble starts when you let them reach into each other, because then a routine app deploy can be blocked behind an infrastructure change, and an infrastructure change can take applications down with it. The fix is not a better tool. It is a thin contract between the two, so each stays in its own lane. I used this to compress a three-environment EKS migration from a planned six months to about three. Here is the contract, and what it actually bought.

The coupling you do not notice until it hurts

The common anti-pattern is to let Terraform deploy applications (via the Helm or kubectl provider) or to let application manifests carry infrastructure facts hard-coded inside them. Both feel convenient. Both create coupling that only shows its cost later.

When Terraform deploys apps, every application change rides the Terraform lifecycle: a terraform apply that should only touch a node group now also reconciles your workloads, and a drift in one app can fail a plan that was supposed to be about infrastructure. When app manifests hard-code infrastructure facts (the cluster's OIDC issuer, a role ARN, a VPC ID, the region), those manifests are no longer portable across environments, and every environment promotion becomes a hand-edit. The blast radius of each change grows to cover both layers, and the audit trail blurs, because you can no longer tell from the change history whether a given commit was an infrastructure decision or an application one.

The two layers genuinely have different natures. Infrastructure changes are infrequent, high-risk, and want slow, careful approval. Application changes are frequent, lower-risk, and want to flow. Forcing them through one pipeline makes the fast side as slow as the slow side, and the slow side as risky as the fast side.

The contract

The bridge is deliberately small. Terraform's job ends at provisioning the cluster and then publishing a set of facts about it into a known location. In practice that location is a ConfigMap and a Secret in a known namespace, carrying labels and annotations that describe the cluster: its name, region, the OIDC provider URL, role ARNs for workloads, networking identifiers, and any environment-specific values the apps will need. Terraform owns writing these. It never installs an application.

ArgoCD's job begins by reading that interface. A root application points at the known ConfigMap/Secret, pulls the cluster facts, and uses them to parameterize everything it deploys. The app manifests themselves carry no hard-coded infrastructure values. They carry references into the bridge. ArgoCD owns reading these. It never provisions infrastructure.

The important property is what is absent. Terraform does not read ArgoCD's state, and ArgoCD does not read Terraform's state. There is no shared state file, no circular dependency where each waits on the other. The only thing they share is the interface, and the interface is data, not control. Terraform writes facts; ArgoCD reads facts. That one-directional, data-only handoff is the whole design, and it is what lets each side keep its own clock.

Why this made a hard migration tractable

The migration that proved it out had a constraint that is more common than people admit: I could not directly touch the target environments. The work had to be delivered as infrastructure-as-code that the customer ran themselves, across three environments, an internal lab, a sandbox, and production, with no interactive access to any of them.

That constraint is exactly where the bridge pays off. Because the apps consume cluster facts through the interface rather than embedding them, the same application definitions deploy unchanged to all three environments. Each environment's Terraform writes its own values into its own bridge, and the GitOps layer adapts automatically. There was no per-environment hand-editing of manifests, which is where this kind of migration usually loses weeks and acquires drift. The customer could run the IaC in each environment and get a reproducible result, because the only thing that varied between environments lived in the bridge, generated by that environment's own provisioning.

For one customer, the combination (Terraform plus the bridge plus GitOps, with Karpenter handling node provisioning on the cluster side) took a migration planned for about six months down to roughly three. The halving did not come from any single tool being fast. It came from removing the coupling that would otherwise have forced serial, hand-held, per-environment work. When each environment comes up the same way from the same definitions, you stop paying the integration tax three times.

How to adopt it

Start by drawing the line. List every value your application manifests currently hard-code that is really an infrastructure fact: cluster name, region, OIDC issuer, IAM role ARNs, subnet or security-group IDs, DNS zone names. Those are the contents of your bridge.

Have Terraform write them. After it provisions the cluster, have it create a ConfigMap (for non-secret facts) and a Secret (for anything sensitive) in a known namespace, populated from its outputs. Decide the key names once and treat them as the contract; renaming a bridge key is now a breaking change, which is the right level of ceremony for it.

Have GitOps read them. Point a root ArgoCD (or Flux) application at that namespace, and parameterize your app set from the bridge values rather than from literals. Each environment gets its own bridge from its own Terraform, and the same app definitions follow.

A minimal, generic version of both sides looks like this. First, Terraform publishes the facts after it provisions the cluster (the values are illustrative placeholders, not from any real environment):

# Terraform: write the bridge after the cluster exists.
# The labels are what ArgoCD's ApplicationSet selects on; the data is the facts.
resource "kubernetes_config_map" "cluster_bridge" {
  metadata {
    name      = "cluster-bridge"
    namespace = "argocd"
    labels = {
      "bridge.example.com/environment" = var.environment   # dev | staging | prod
    }
  }

  data = {
    cluster_name   = module.eks.cluster_name
    region         = var.region
    oidc_issuer    = module.eks.oidc_provider          # for IRSA
    account_id     = data.aws_caller_identity.current.account_id
    # a workload's IAM role ARN, resolved per environment:
    app_irsa_role  = module.app_irsa.iam_role_arn
  }
}

Then an ArgoCD ApplicationSet reads that ConfigMap and templates every app from it, so the app definitions themselves carry references, never literals:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-addons
  namespace: argocd
spec:
  generators:
    - clusters:                      # or a configMap/plugin generator reading the bridge
        selector:
          matchLabels:
            bridge.example.com/environment: prod
  template:
    metadata:
      name: '{{name}}-addons'
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/platform-addons
        targetRevision: main
        path: addons
        helm:
          # bridge facts flow in as values; no hard-coded ARNs or issuers in the chart
          parameters:
            - name: region
              value: '{{metadata.annotations.region}}'
            - name: oidcIssuer
              value: '{{metadata.annotations.oidc_issuer}}'
            - name: appIrsaRole
              value: '{{metadata.annotations.app_irsa_role}}'
      destination:
        server: '{{server}}'
        namespace: kube-system

The exact generator you use (cluster, configMap, or a plugin generator) depends on your setup; the invariant is the one that matters: Terraform owns the left side and writes facts, ArgoCD owns the right side and reads them, and the app charts reference bridge values instead of embedding them. Swap prod for dev or staging and nothing in the app definitions changes, only the bridge each environment's Terraform wrote.

Hold the boundary. The one discipline that keeps this working is refusing to cross the line later under deadline pressure: no Helm-provider app installs sneaking into Terraform, no infrastructure ARNs sneaking back into manifests. The value is entirely in the separation, so the separation is the thing to protect.

The reward is that your fast layer stays fast and your slow layer stays safe. An application deploy no longer waits on an infrastructure plan. An infrastructure change no longer reconciles your workloads as a side effect. And a new environment is a new bridge, not a new round of manual edits. Keep the two clocks separate, and let a thin, one-way, data-only contract be the only thing between them.

If you run Terraform and a GitOps controller together, it is worth asking which infrastructure facts your app manifests still hard-code. Those are the ones the bridge is for.

The reality of GitOps application recreation

Tue, 21 Apr 2026 04:19:42 +0000

Your application code is in Git, and you've adopted GitOps principles, so you can recreate it anywhere, anytime, right? If you're like many teams, the honest answer is "sort of". While GitOps promises the ability to recreate your application from version control, the reality is often more nuanced as you consider the holistic view of deploying and running your applications.

Our State of GitOps report reveals some fascinating insights about recreatable applications. As part of the survey, we asked this question:

Can you recreate the application from the configuration files stored in version control, for example, to recover from a disaster or to create a new test environment?

Overall, 52% of respondents feel confident they can recreate their applications from version control. For high-performing GitOps teams, this jumps to 70%. But what caught my attention is that "partially recreatable" was a significant response across GitOps maturity levels and the most common response among teams still developing their GitOps practices. It suggests that while most teams can handle their core applications, they struggle with recreating the complete environment stack.

Interestingly, our data reveals that teams often become less confident about recreation as they advance their GitOps practices, only to see confidence surge at higher maturity levels. This pattern likely reflects a natural discovery process. As teams dive deeper into GitOps, they uncover dependencies and complexities they didn't initially realize existed.

So, what does complete application recreation entail? And why do so many teams find themselves in that "partially recreatable" category?

The real-world value of recreatable applications

Recreatable applications don't make sense for all organizations and applications and arguably could be viewed as nice to have rather than a necessity. However, recreating your applications from Git can solve real problems teams face at different scales and circumstances.

When disasters occur, having your entire application stack defined in code means restoration becomes a deployment rather than a scramble. You can rebuild from your Git repository instead of hunting through documentation or recreating manual configurations. The same capability proves valuable during cloud provider migrations driven by acquisitions, regulatory requirements, or strategic business decisions, where recreation becomes a non-trivial but controlled transition rather than a risky lift-and-shift operation.

Of course, your application is just one piece of the puzzle. You'll still need to consider the surrounding infrastructure, networking, and foundational services that enable your application to run in the target environment. While mature teams often manage this infrastructure through tools like Terraform and Crossplane, getting to that level of complete recreation from Git requires thoughtful planning and infrastructure provisioning processes.

Operational efficiency improves when you can create new test environments on demand with minimal overhead. Whether you're testing critical fixes, running performance tests against production-like infrastructure, or validating new features, the ability to spin up identical environments quickly and tear them down when finished reduces both time and infrastructure costs.

Recreation provides auditable proof for regulated industries that infrastructure and deployment processes are fully documented and reproducible. Suppose you need to satisfy compliance frameworks that require demonstrable change control. In that case, recreatable applications help you meet audit requirements for deployment consistency and provide evidence that you can rebuild systems according to documented specifications.

The maturity journey

Our survey data's confidence curve tells a story about how teams learn and adopt GitOps. Rather than steady upward progress, there's an initial dip in confidence before teams reach far higher levels of certainty about their recreation capabilities.

This pattern might highlight the natural learning process. Pre and low adopters will typically approach GitOps from an application-down perspective and focus on getting their manifests into Git repositories. The initial confidence may come from successfully deploying applications this way and feeling like they've "solved" recreation.

However, as teams mature, reality sets in when they add more applications to their GitOps processes and try to recreate complete environments. They likely discover their core applications are relatively easy to recreate, but the surrounding environment may not be. Infrastructure provisioning may be required, whether manual or automated, and teams may not have accounted for external dependencies yet.

You need backup and restore strategies for stateful components like databases that go beyond what you define in Kubernetes manifests. While the application might be recreatable from Git, the data likely isn't. You’ll consider whether to increase cost and complexity by replicating databases across infrastructure, configure automated backup and restore processes, or accept that provisioning new environments require data restoration as a separate step.

This explains why "partially recreatable" was a popular response in our survey across all GitOps maturity levels. Most teams can handle their core applications but struggle with the complete environment stack. As low and medium-maturity teams adopt foundational GitOps practices, they discover the full scope of what requires management, decreasing their confidence in complete recreation.

What to consider with complete recreation

Success stories do exist. Teams using infrastructure provisioning tools like Crossplane, Terraform, and mature GitOps workflows have achieved recreatable applications. But getting there requires stepping back and considering what recreation means from a holistic perspective.

Your application manifests assume that the underlying infrastructure exists, but something must first create that foundation. Many organizations still manage infrastructure provisioning through separate Terraform workflows, creating a gap where recreation often breaks down. While infrastructure provisioning may be mostly automated, unless integrated with your GitOps workflows, you must step away and use another platform to create the infrastructure first.

Recreation means recreating the configuration and access to secrets in the new environment. Your GitOps process needs strategies for safely managing environment-specific values, API keys, and certificates.

Teams can easily recreate applications but can't recreate data the same way. You need strategies for database backups, data replication, or accepting that data restoration happens separately from application recreation.

External services, APIs, or legacy systems your applications depend on often fall outside what GitOps can recreate. Your recreation strategy needs to account for these dependencies, whether through service discovery, configuration updates, or fallback mechanisms. Additionally, can your target environment support the infrastructure you rely on? Not all cloud services are available in every region, especially during disasters.

70% of high-performers have achieved confidence in their recreation capabilities, and have worked through these considerations. They prove that complete recreation is possible and likely find it worthwhile, but it requires treating it as a comprehensive system design challenge rather than just putting YAML in Git.

The path forward

While the journey from partial to complete recreation involves discovering complexities you didn't know existed, 70% of high-performing teams that achieved this capability prove it's possible and worthwhile.

The key is treating recreation as a systematic challenge rather than an afterthought. Whether you're just starting your GitOps journey or working through the complexities that come with maturity, understanding what complete recreation entails helps you make informed decisions about where to invest your efforts.

For more insights into how teams across different maturity levels approach GitOps and application recreation, download our complete State of GitOps report to see the full research findings and implementation patterns.

Beyond InferenceService Readiness: 5 GitOps Failure Modes That Break KServe Deployments

Fri, 17 Apr 2026 11:54:28 +0000

A sequel to my KServe readiness post — five GitOps control-plane failure modes with exact terminal output, diagnostics, and repeatable fixes for ArgoCD + KServe stacks.

This post is a follow-up to my earlier KServe piece on endpoint readiness:

Why Your KServe InferenceService Won't Become Ready: Four Production Failures and Fixes

That article focused on why an InferenceService may not become Ready.

This one zooms out to a broader question:

What breaks when the GitOps control plane itself is unstable?

Most GitOps + AI serving tutorials still focus on the happy path — install ArgoCD, apply KServe, deploy InferenceService, done. But in real platform work, the happy path is the easy part.

The hard part is when your app is OutOfSync, the webhook has no endpoints, and everything looks healthy except the thing you actually need.

This post covers the five failure modes that repeatedly broke KServe deployments in a real production-grade platform build, with exact terminal output, root causes, and the fixes that worked.

All failures come from hands-on implementation work documented here:
Project repo: github.com/sodiq-code/neuroscale-platform

The platform context

Stack:

ArgoCD — GitOps reconciliation
KServe — model serving (InferenceService, runtimes)
Knative + Kourier — serving networking
Kyverno — policy guardrails
Backstage — self-service PR generation

GitOps root app:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: neuroscale-infrastructure
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/sodiq-code/neuroscale-platform.git
    targetRevision: main
    path: infrastructure/apps

Failure Mode 1: Webhook Has No Endpoints — Sync Fails Cluster-Wide

Time lost: ~1 hour | Impact: All InferenceService operations blocked

Symptom

ArgoCD syncs child apps and hits this:

$ kubectl -n argocd describe application ai-model-alpha
...
Message: admission webhook
  "inferenceservice.kserve-webhook-server.validator.webhook"
  denied the request: Internal error occurred:
  no endpoints available for service "kserve-webhook-server-service"

Meanwhile the KServe controller pod shows only 1 of 2 containers ready:

$ kubectl -n kserve get pods
NAME                                        READY   STATUS
kserve-controller-manager-8d7c5b9f4-xr2lm  1/2     Running

$ kubectl -n kserve describe pod kserve-controller-manager-8d7c5b9f4-xr2lm
...
  kube-rbac-proxy:
    State:   Waiting
    Reason:  ImagePullBackOff
    Image:   gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1
Events:
  Warning  Failed  kubelet
    Failed to pull image: unexpected status code 403 Forbidden

Root Cause

The kube-rbac-proxy sidecar inside kserve-controller-manager was pulling from gcr.io/kubebuilder/ — a registry that restricted access in late 2025. The manager container was healthy but because the sidecar was not running, the webhook server had no valid certificate endpoint. Result: every InferenceService apply or update was blocked cluster-wide.

Fix

Remove the sidecar via Kustomize strategic merge patch:

# infrastructure/serving-stack/patches/
#   kserve-controller-kube-rbac-proxy-image.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kserve-controller-manager
  namespace: kserve
spec:
  template:
    spec:
      containers:
        - name: kube-rbac-proxy
          $patch: delete

Verify webhook endpoints are restored after re-sync:

$ kubectl -n kserve get endpoints kserve-webhook-server-service
NAME                           ENDPOINTS          AGE
kserve-webhook-server-service  10.42.0.23:9443    45s

Lesson

When webhook endpoints are missing, your app YAML is never the real problem. Diagnose the controller first. An external registry access change can silently kill your entire admission layer cluster-wide with no obvious error in the app itself.

Failure Mode 2: CRD Deleted by a Misapplied Patch — All Endpoints Gone Instantly

Time lost: 4 minutes recovery | Impact: SEV-1 equivalent — all InferenceServices deleted

Symptom

All InferenceService objects disappeared silently:

$ kubectl -n default get inferenceservices
No resources found in default namespace.

$ kubectl -n argocd get application demo-iris-2
NAME          SYNC STATUS   HEALTH STATUS
demo-iris-2   OutOfSync      Missing

Root Cause

A Kustomize patch file named remove-inferenceservice-crd.yaml was mistakenly applied directly with kubectl apply -f instead of being used as a build-time patch inside kustomization.yaml. The file contained a $patch: delete directive:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: inferenceservices.serving.kserve.io
$patch: delete

When applied directly, it deleted the actual CRD from Kubernetes. When a CRD is deleted, Kubernetes immediately garbage-collects every custom resource of that type. Every InferenceService was gone within seconds.

Fix

Restore the CRD immediately:

kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.12.1/kserve.yaml

kubectl wait crd/inferenceservices.serving.kserve.io \
  --for=condition=Established --timeout=60s

kubectl -n argocd patch application demo-iris-2 \
  --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

Lesson

$patch: delete in a Kustomize file is a build-time instruction — it tells kustomize build to omit that resource from output. It must never be applied directly with kubectl apply -f. Ambiguous filenames like remove-inferenceservice-crd.yaml are dangerous footguns. In a production cluster with 50 deployed models this would be a full SEV-1.

⚠️ Rule: Any file containing $patch: delete must only ever be referenced inside a kustomization.yaml patches block, never applied directly.

Failure Mode 3: Permanent OutOfSync Due to Label Key Mismatch

Time lost: 2 weeks undetected | Impact: CI was green while policy enforcement was silently broken

Symptom

A PR is merged, ArgoCD syncs, but the InferenceService stays OutOfSync/Degraded:

$ kubectl -n argocd get application demo-iris-2
NAME          SYNC STATUS   HEALTH STATUS
demo-iris-2   OutOfSync      Degraded

Kyverno denies the resource at admission:

Error from server: error when creating "STDIN":
  admission webhook "clusterpolice.kyverno.svc" denied the request:
  resource InferenceService/default/test-model was blocked due to the following policies
  require-standard-labels-inferenceservice:
    check-owner-and-cost-center-on-isvc: 'validation error:
    InferenceService resources must set metadata.labels.owner and
    metadata.labels.cost-center.
    rule check-owner-and-cost-center-on-isvc failed at path
    /metadata/labels/cost-center/'

But the label is present in the manifest:

$ kubectl -n default get inferenceservice demo-iris-2 \
    -o jsonpath='{.metadata.labels}' | python3 -m json.tool
{
    "owner": "platform-team",
    "costCenter": "ai-platform"
}

Root Cause

costCenter (camelCase) and cost-center (kebab-case) are completely different Kubernetes label keys. The Backstage template skeleton was generating costCenter. The Kyverno policy required cost-center. CI passed because CI used the same manifest that would pass — the mismatch only surfaced at admission time.

Additionally, kyverno-cli apply exits with code 0 even when policy violations are found. CI was checking $? rather than ${PIPESTATUS[0]}, so the CI step appeared green while enforcement was completely broken for two weeks.

Fix

Standardize on kebab-case throughout (Kubernetes convention):

# Backstage template skeleton
# apps/${{ values.name }}/inference-service.yaml
labels:
  owner: platform-team
  cost-center: ai-platform   # was: costCenter

Fix the CI Kyverno check to catch actual violations:

set +e
docker run --rm -v "$PWD:/work" -w /work ghcr.io/kyverno/kyverno-cli:v1.12.5 \
  apply infrastructure/kyverno/policies/*.yaml \
  --resource "${app_files[@]}" \
  2>&1 | tee /tmp/kyverno-output.txt
kyverno_exit="${PIPESTATUS[0]}"
set -e

if [ "${kyverno_exit}" -ne 0 ] \
    || grep -qE "^FAIL" /tmp/kyverno-output.txt \
    || grep -qE "fail: [1-9][0-9]*" /tmp/kyverno-output.txt; then
  echo "Kyverno policy violations detected. Failing CI."
  exit 1
fi

Lesson

$? captures the exit code of tee, not kyverno. ${PIPESTATUS[0]} captures kyverno's actual exit code. "Guardrails exist" and "guardrails enforce" are different states. The most dangerous failure mode for a policy system is silent false positives — everything looks green while nothing is actually being enforced.

Failure Mode 4: Kyverno Install Breaks ArgoCD Reconciliation Loop

Time lost: 2–5 minutes per cluster | Impact: All ArgoCD apps enter Unknown state

Symptom

After adding Kyverno to the platform, previously healthy apps enter Unknown state:

$ kubectl -n argocd get applications
NAME                       SYNC STATUS   HEALTH STATUS
neuroscale-infrastructure  Synced         Healthy
serving-stack              Unknown        Unknown    # was Healthy 10 minutes ago
policy-guardrails          Synced         Healthy

$ kubectl -n argocd describe application serving-stack
...
Message: rpc error: code = Unavailable desc = connection refused

Root Cause

Kyverno installs its own ValidatingWebhookConfiguration and MutatingWebhookConfiguration during install. While Kyverno is initializing, the webhook configurations are registered but point to endpoints that do not exist yet. During this window, any kubectl apply operation — including ArgoCD's sync reconciliation loop — times out waiting for a response from a not-yet-running webhook server. This cascades into the ArgoCD repo-server losing its connection.

Fix

Add a Kyverno webhookAnnotations ConfigMap patch to suppress automatic webhook registration during the initialization window:

# infrastructure/kyverno/kustomization.yaml
patches:
  - target:
      kind: ConfigMap
      name: kyverno
    patch: |-
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: kyverno
        namespace: kyverno
      data:
        webhookAnnotations: "{}"

After Kyverno reaches Running state, force a hard refresh:

kubectl -n argocd patch application serving-stack \
  --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

Lesson

Adding a policy engine to an existing cluster disrupts all other ArgoCD-managed applications during the install window. In production, this requires a maintenance window or a canary install strategy. Kyverno must be fully healthy before any other component syncs.

Failure Mode 5: Stale Admission Webhook Blocks All Resource Creation

Time lost: 30+ minutes | Impact: All Deployments in the namespace silently blocked

Symptom

After fixing the repo-server, apps sync but Deployments never appear:

$ kubectl get applications -n argocd
NAME                       SYNC STATUS   HEALTH STATUS
neuroscale-infrastructure  Synced         Healthy
test-app                   Synced         Progressing   # stuck

$ kubectl get deploy -n default
No resources found in default namespace.

ArgoCD shows the Deployment as "synced" but it does not exist — a contradiction. Checking conditions:

$ kubectl -n argocd get application test-app -o yaml | grep -A 20 conditions
  conditions:
  - message: 'Failed sync attempt: one or more objects failed to apply,
      reason: Internal error occurred: failed calling webhook
      "validate.nginx.ingress.kubernetes.io":
      failed to call webhook: Post
      "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/...":
      dial tcp 10.96.x.x:443: connect: connection refused'
    type: SyncError

Root Cause

A ValidatingWebhookConfiguration from a previous cluster experiment was still registered but pointing to a service that no longer existed. Kubernetes admission webhooks are cluster-scoped. The stale ingress-nginx webhook was intercepting every resource creation attempt and failing them — the error only appears in ArgoCD events, not on the Deployment itself.

Fix

# Discover stale webhooks
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

# Delete the stale one
kubectl delete validatingwebhookconfiguration ingress-nginx-admission

# Force ArgoCD to retry
kubectl -n argocd patch application test-app \
  --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

After deletion:

$ kubectl get deploy -n default
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
nginx-test    1/1     1            1           23s

Lesson

A stale webhook from a previous workload silently blocks all resource creation in the affected namespace for hours without any obvious error message. The admission error only appears in ArgoCD events logs, not on the resource itself. Always check for stale webhooks before blaming manifests.

The Triage Sequence That Saves Hours

When a KServe app is failing in ArgoCD, run this exact order before touching any manifest:

# 1. Environment gate — if this fails, stop and fix environment first
kubectl get nodes
kubectl -n argocd get applications

# 2. Control-plane health
kubectl -n kserve get deploy,pods,svc,endpoints
kubectl get crd | grep serving.kserve.io

# 3. Controller logs
kubectl -n kserve logs deploy/kserve-controller-manager --tail=100

# 4. Webhook availability
kubectl -n kserve get endpoints kserve-webhook-server-service

# 5. Stale webhooks
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

# 6. App-level sync error detail
kubectl -n argocd get application <app-name> -o yaml | grep -A 20 conditions

Only after every step above passes should you edit app manifests.

Why This Matters for Platform Teams

A platform is credible when it supports both:

Self-service delivery — the Golden Path works
Self-service recovery — failures are understandable and fixable without a platform expert

Most teams build the first and postpone the second. That creates operational debt fast.

The fix is not more dashboards. It is better failure-model documentation, tighter GitOps guardrails, and the discipline to document what breaks — not just what works.

A platform is not "done" when the happy path works. It's done when the failure path is understandable and recoverable.

What I Would Improve Next

Pre-merge CI assertions for probe and resource fields in rendered manifests
Explicit dependency ordering using ArgoCD sync waves to prevent Kyverno install disruption
Conformance checks for Helm dependency values nesting to catch silently ignored overrides
Policy test fixtures that verify both pass and fail cases in CI

ArgoCon Lightning Keynote: Argo CD vs the world

Tue, 31 Mar 2026 09:34:08 +0000

An exclusive look at our research data to see how Argo CD stacks up against everything else when it comes to GitOps and software delivery.

6 GitOps Practices That Actually Work

Fri, 27 Feb 2026 14:16:00 +0000

GitOps was pitched as a transformative approach to software delivery and infrastructure management, promising better outcomes across multiple dimensions of organizational performance. But, not all GitOps implementations are created equal.

Research from the State of GitOps report reveals six crucial practices statistically related to several meaningful outcomes, including improved software delivery, enhanced reliability, eliminating configuration drift, and easier compliance and audit.

These practices form the foundation of successful GitOps adoption, moving beyond superficial implementation to achieve meaningful business outcomes. Organizations missing one or more of these practices were less likely to gain the benefits of GitOps.

1. Declarative Desired State

Unlike imperative approaches that require step-by-step instructions, declarative configuration describes the end state you want to achieve, leaving the “how” to automated reconciliation tools. This represents a shift from crafting scripts that call a sequence of APIs and commands to drive state.

This shift from imperative to declarative thinking addresses one of the most common sources of operational complexity: the mental overhead of understanding how a series of changes will affect system state. This is difficult enough if you assume you know the baseline state and have tight controls to prevent drift, but these assumptions are often incorrect.

Declarative desired state eliminates this complexity by providing a clear, verifiable target. Instead of worrying about the sequence of operations, teams can focus on defining what they want the system to look like.

What you get from declarative desired state:

A readable change history and audit trail.
Transfer of reconciliation burden to tools.
A self-documenting target state.

2. Human Readable Format Is Critical

The power of version control for change management only materializes when the configuration files are human-readable. This practice might seem obvious, but many organizations undermine their GitOps effectiveness by obscuring meaning with binary formats, encoded configurations, or overly complex and verbose markup.

Human-readable formats prioritize readability and comprehension, making it easier to make a change, review changes, or understand the target state. When files aren’t human-readable, making changes is harder, code reviews take longer, and nobody can understand the intended system state. You get a second wave of pain when you need to review an audit trail or determine what change introduced a problem.

What you get from human-readable formats:

Easier changes that follow the developer-style workflow.
Faster and easier code reviews.
Change history and audit trails from version control.

3. Responsive Code Review

The speed of code review in GitOps implementations directly impacts the practice’s effectiveness and the likelihood of its consistent adoption. Slow reviews create a cascade of adverse effects that can undermine the entire GitOps approach.

When reviews are sluggish, teams naturally batch changes to reduce the frequency of the review bottleneck. These larger batches increase complexity, making reviews more difficult and time-consuming, perpetuating the cycle. More critically, large batches increase the risk of each deployment and make it harder to isolate issues when problems occur.

Slow reviews also create pressure to bypass the GitOps process. Teams may resort to direct modifications that circumvent version control when they face urgent issues or tight deadlines. These “emergency” changes break the audit trail and create configuration drift, eroding the benefits of GitOps adoption.

What you get from responsive code reviews:

Smaller batches have less risk.
The ability to make version control the primary interface for system changes.
Better throughput as people are no longer blocked waiting for a review.

4. Version Control

While everyone understands version control is a fundamental component of GitOps, its effectiveness depends heavily on the preceding practices. Version control becomes a powerful platform for change management only when you use human-readable declarative files and review changes quickly.

The selection of version control as the foundation for GitOps allows the reuse of existing organizational practices for access control, backup, and disaster recovery. Organizations typically have mature processes for protecting and managing their source code repositories, and you can easily apply these same protections to configuration repositories.

What you get from version control:

A source of truth for the desired state of the system.
Familiar tools, controls, and audit trails.
A complete history of changes and mechanisms to roll back changes.

5. Automatic Pull

With automatic pull, both words are equally important. The automation is crucial because you want to keep the system in the desired state. Pull is similarly essential as GitOps shifts from a central orchestrator pushing out changes to a distributed set of agents. This makes it easier to scale as you add more infrastructure, as you don’t need to maintain a central list.

The pull model also aligns with cloud-native architecture patterns, where services are expected to be self-contained and resilient to external dependencies. This architectural alignment makes GitOps a natural fit for modern application platforms and container orchestration systems, though it’s also used beyond Kubernetes.

What you get from automatic pull:

Increased security as you don’t need to expose endpoints.
Simpler fleet management, as you don’t need to manage the list of destinations centrally.

6. Continuous Reconciliation

Continuous reconciliation involves automatically detecting and correcting deviations from the desired state. The reconciliation loop is core to GitOps, but many organizations haven’t implemented it.

If you’re serious about eliminating configuration drift, you must implement continuous reconciliation. Even with the best will in the world, your chances of preventing drift are low unless you automatically and frequently return the system to the target state.

What you get from continuous reconciliation:

Confidence that the system remains intended.
Motivation for all changes to be made through version control.

It’s Not Gatekeeping, It’s Data

The research backing these practices provides quantitative evidence that GitOps delivers measurable business value if you adopt the proper practices. Organizations considering.

Success in GitOps comes not from a perfect initial implementation but from continuous improvement guided by these research-backed practices. Teams that start with basic implementations of all six practices achieve better outcomes than those that perfect one or two practices without adopting the rest.

Automation is the best way to manage complexity in software systems. The six practices of GitOps provide a model for this that has been proven to work.

You can find more about the research in the State of GitOps report, complete the quick GitOps assessment, and watch on demand The New Stack’s webinar, recorded July 17, 2025.

Autopilot - the Final: From Localhost to the Cloud with GitHub Actions and CI/CD

Mon, 16 Feb 2026 06:54:01 +0000

We have come a long way. We started by designing a Brain capable of reading (Post 2), we gave it personality with Creative Agents (Post 3), and we fought against bureaucracy to get some Hands (APIs) that could publish legally (Post 4).

But there was one last big step left so as not to be a slave to my terminal. Right now, to publish, I had to be at my computer, open the console, and run python main.py. That's not "Autopilot." That's "Assisted Driving."

Today, in the final chapter, we cut the cords. We move to the cloud and automate the entire process with my AI agents.

The Pipeline Architecture

The goal is GitOps: that my only interaction with the system is pushing changes to Git. Everything else must happen by magic (or rather, by GitHub Actions).

I have designed a two-phase workflow:

Detection and Preview Phase (Automatic):
- GitHub detects a new .md file (or changes to an existing one).
- The Orchestrator activates.
- The system detects the post language (Spanish/English) and calculates the correct URL.
- The AI (or template system) proposes a tweet and a LinkedIn post.
- The system shows me a "Preview" in the execution logs, but does not publish anything.
Publishing Phase (Manual):
- The process pauses automatically thanks to GitHub Environments.
- I receive an alert to review the deployment.
- If I click the green button (Approve), the system executes the real call to the APIs.

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#f0f4f8', 'edgeLabelBackground':'#ffffff', 'tertiaryColor': '#e6e6e6'}}}%%
graph TD
    %% Initial Node
    START([GitHub detects change in .md]) --> ORC

    %% --- PHASE 1: AUTOMATIC ---
    subgraph Phase1 [" Phase 1: Detection and Preview (Automatic)"]
        direction TB
        ORC[Orchestrator Activates]
        
        %% Parallel Tasks
        ORC --> TASK1[Detect Language and Calculate URL]
        ORC --> TASK2[AI proposes Tweet and LinkedIn]
        
        %% Convergence
        TASK1 --> LOGS
        TASK2 --> LOGS
        
        LOGS[Show 'Preview' in Execution Logs]
        LOGS --> NOPUB[ NOTHING PUBLISHED YET]
    end

    NOPUB --> PAUSE

    %% --- PHASE 2: MANUAL ---
    subgraph Phase2 [" Phase 2: Publishing (Manual)"]
        direction TB
        PAUSE((⏸️ AUTOMATIC PAUSE<br/>GitHub Environments))
        
        PAUSE --> ALERT[ Alert arrives to review deployment]
        ALERT --> DECISION{Approve Deployment?}
        
        %% Approval Path
        DECISION -- "Green Button (Approve) ✅" --> EXEC[ Execute real API call]
        
        %% Rejection Path (Implicit)
        DECISION -- "Reject / Cancel ❌" --> STOP([End of flow without publishing])
    end

    %% Styles to highlight final steps
    style EXEC fill:#d4edda,stroke:#28a745,stroke-width:2px,color:#155724
    style STOP fill:#f8d7da,stroke:#dc3545,stroke-width:2px,color:#721c24
    style PAUSE fill:#fff3cd,stroke:#ffc107,stroke-width:3px

The Orchestrator (`orchestrator.py`)

I needed a script to join all the pieces together. To do this, I started developing a Python orchestrator that acts as a bridge between the Markdown file and my social media modules.

This script is in charge of the "fine" logic that we sometimes forget:

Is it a post in English (/en/) or Spanish (/es/)?
Does it have a featured image to generate the Twitter/X or LinkedIn card?
Do I want the AI to write it, or do I want to write it myself?

The Star Feature: "Director's Cut"

Sometimes, the AI doesn't get the tone exactly right, or I simply want to write the copy myself for a special announcement. To not lose automation but maintain control, I implemented a "Manual Overwrite" logic using Hugo's Frontmatter.

If my script detects this in the article header:

---
title: "My Great Post"
social_text: "Today I don't want the AI to write for me. This post is so special that I hand-wrote this. "
---

The system ignores the automatic generation and uses my exact words. It is the perfect balance: automation by default, manual control when necessary.

Security and CI/CD: Sleeping Soundly

The .github/workflows/autopilot.yml file is where the magic happens. Here we define the "Secrets" (my Twitter and LinkedIn API keys) and the rules of the game.

The most interesting part is the environment protection:

jobs:
  publish:
    environment: production  # <--- The key to security
    needs: check_changes
    steps:
      - run: python orchestrator.py

By defining the environment as production, GitHub forces me to review and approve the deployment. This prevents a code error or an AI "hallucination" from publishing unwanted content.

Additionally, we have configured the system so that Twitter/X generates the Cards with images automatically and LinkedIn treats the content as an "Article," ensuring that on both networks the blog's featured image looks large and attractive.

The Final Result

Now, my publishing process is this:

I write my article in Markdown peacefully.
I run git push.
I have a coffee. ☕
I check GitHub from my phone, see the "Preview" of the generated tweet (in the correct language).
I smile and hit Approve.

In seconds, the content appears on Twitter and LinkedIn. Without opening the terminal. Without touching Python. From anywhere.

Twitter/X Publication

LinkedIn Publication

Autopilot Project Conclusion

What started as an experiment to test how AI agents work and how they can help in my day-to-day life has turned into a professional publishing system. We have touched upon:

Prompt Engineering to define personalities.
Complex OAuth 2.0 APIs and token management.
Robust Python backend.
DevOps and CI/CD with GitHub Actions.

This blog is no longer just a collection of texts; it is a living application that works for me. And now that I have free time... what shall we automate next?

Thanks for joining me in this series.

Final Source Code: The entire project is available (and documented) on GitHub.

SHIPPED25: The Research Panel

Tue, 09 Dec 2025 21:43:05 +0000

In this panel discussion, Matt Allford, Charlotte Fleming, and Steve Fenton walk through some of the key findings from the research projects that took place at Octopus Deploy this year.

The State of GitOps report

The State of GitOps report investigates the adoption, practices, and benefits of a GitOps approach. Our research is based on data from 660 survey responses and interviews with a panel of experts and practitioners.

The Platform Engineering Pulse report

The Platform Engineering Pulse report reveals how organizations implement Platform Engineering today, what's working, what isn't, and why. We surveyed technical professionals across industries and roles to understand the gap between Platform Engineering theory and reality. The research explores what drives adoption decisions, which platform features deliver real value, how teams measure success, and which strategies actually move the needle.

Compliance through Continuous Delivery

This report, surveying 50 financial organizations in 2025, shows how automated Continuous Delivery pipelines can simultaneously increase throughput and stability, resolving the speed-versus-safety trade-off.

GitOps Gap: Few Use Declarative Configuration To Manage State

Tue, 11 Nov 2025 10:18:38 +0000

Given the advantages of declarative desired state and the tools that reconcile it, why aren’t organizations adopting it faster?

Declarative desired state is the GitOps principle everyone knows but few actually use. The recent State of GitOps report found that just 40% of organizations were using declarative configuration, so what is it and why is it so fundamental to successful GitOps?

What Is Declarative Desired State?

There are two ways to manage state in GitOps. The first is to define a sequence of steps to take a baseline and move it into the desired state. This checklist-style approach is imperative: a set of step-by-step instructions reach the intended outcome. In contrast, declarative state defines the end state, not the steps to reach it.

We can think of this in terms of pizza. You can follow the recipe by stretching out a pizza base, then adding tomato sauce, then topping it with cheese and a couple of tomato slices. Alternatively, you can show an experienced pizza chef a picture of a Margherita and they will take care of everything. Following the recipe gets you an imperative pizza, while showing the picture gets you declarative pizza.

In the same way, imperative configuration contains a series of API calls, commands and scripts, whereas declarative configuration defines the target state and moves the heavy lifting to the tools.

Open GitOps says: “A system managed by GitOps must have its desired state expressed declaratively.” There are only four principles and this is the first of them, so it must be important. But why?

Why Declarative Desired State Is Crucial

Declarative configuration provides a verifiable target state. That makes it useful beyond use within a tool, as it is documentation of what the system should look like. To build the same understanding with imperative configuration, you’d have to run through all the steps and understand how they would change a baseline state, which means you need to hold the whole process in your mental model to understand the end result. This also leaves room for assumptions and omissions, and may result in a state that isn’t what you imagined.

Declarative configuration is simpler to understand. It pushes the burden of achieving that state onto automation tools.

With declarative desired state you get:

A documented target state
Easier changes and reviews
A readable change history and audit trail
Transfer of reconciliation burden to tools

Given the advantages of declarative desired state and the tools that reconcile it, why aren’t organizations adopting it faster?

Not a Simple Task

While it’s easy enough to start a new project with declarative desired state, there are factors that make it difficult to switch an existing system.

The longer a system exists before you bring it under the control of automatic reconciliation, the longer it is exposed to ClickOps and configuration drift. This makes it hard or even impossible to understand and document the desired state.

Older systems may not be independent or modular, and it may be difficult to build a picture of what the desired state ought to be. Past decisions may force imperative configuration — for example, if they require chained deployments where services must be updated in a specific order.

If you’ve automated database schema changes, you’ll have faced a similar challenge. It was often easier to take a baseline backup of your database, then apply scripts to alter the schema from that point on.

Modern database tools make it possible to retrospectively generate these scripts from a model of the schema’s desired state. GitOps tools that can generate the initial declarative desired state from running systems could perform a similar role in improving the adoption of this practice.

Roles and Regions Are on a Journey

Despite the overall low adoption rate for declarative desired state, some job roles are ahead of the curve in terms of adoption. Architects are the most likely to use declarative configuration, with 60% of them using it. System administrators are the furthest behind at 33.3%.

There are also regional differences, with Europe, North America and Oceania each having more than 40% adoption, while other regions have less than 30%.

All these numbers are likely to improve as organizations grow and mature their GitOps practices. Those who are just starting out with GitOps will see improvements from other GitOps practices regardless of their choice of declarative or imperative configuration. Once they reach a higher level of maturity, declarative configuration will eventually be the place to look for further improvements.

Declarative Configuration Will Rise

Declarative configuration is one of six practices that complement and amplify each other. While you’ll only get the full benefit of GitOps by adopting all the recommended practices, you can work on introducing these over time so you can effectively onboard them and develop a high degree of skill and confidence before adding more. The six practices and principles are:

Declarative desired state
Human readable format
Responsive code review
Version control
Automatic pull
Continuous reconciliation

The State of GitOps report indicates many organizations are starting out on their GitOps journey. As they gain the skills and experience to apply these principles and practices, you can expect the number of organizations using declarative configuration to increase. The benefits are too important to ignore.

You can find out more by downloading the report and by watching the Octopus State of GitOps report webinar.

The GitOps maturity model

Fri, 24 Oct 2025 12:01:56 +0000

We love maturity models, but you must take great care when deciding whether they are the appropriate tool for the job. In DevOps, the volume of practices and the number and variety of contexts make maturity models a poor fit. It’s far better to review all available options within your context when deciding how to deliver software better.

GitOps is different. Strong agreement exists on the handful of principles and practices. The State of GitOps research also confirms that the practices are mutually supportive, so you must use them in concert, not alone.

Based on the research, we can create a maturity model inspired by the great work done in the past but updated based on statistical analysis of what organizations are doing and what works.

This new GitOps maturity model provides an evidence-backed path from early foundations to operating at scale.

The current version of the GitOps maturity model is: v2025.1.

The statistical model

Based on the research, we build a statistical model of relationships between GitOps practices and many outcomes. The model highlighted where there was a link between a practice and an outcome and also the strength of the relationship.

The statistical analysis revealed several key insights that inform this maturity model:

Practice Dependencies: Certain GitOps practices show strong statistical correlation with others, indicating natural adoption paths for practices
Outcome Correlation: The data demonstrates clear relationships between specific practices and both DevOps and GitOps-specific outcomes
Cumulative Effect: Organizations practicing multiple GitOps practices simultaneously show exponentially better results than those cherry-picking individual practices

We can combine these elements into a simplified maturity model diagram by showing the practices driving the 3 DevOps and 5 GitOps outcomes.

In the model, solid outlines indicate practices based on statistical research. This GitOps maturity model is advantageous as it’s technology and vendor-neutral. While many teams use GitOps for Kubernetes, the research found that a solid percentage of organizations use it for other technology stacks. The maturity model doesn’t recommend specific tools because you’ll select different options based on what you need to deploy and to where.

The research data shows that organizations adopting these practices achieve measurably better outcomes across deployment frequency, lead times, change failure rates, and mean time to recovery compared to those attempting to skip stages or implement practices in isolation.

The maturity model will be updated based on feedback, with future research being used to validate any suggestions.

Pre-adoption

Before adopting GitOps, organizations use familiar workflows with code commits triggering CI/CD events and manually-driven infrastructure updates, either using ClickOps or manually triggering infrastructure scripts.

This leads to some predictable challenges:

Deployment inconsistency: Without a single source of truth, environment configurations drift apart. Production has a different configuration to staging, so there’s no guarantee the application you tested will work when exposed to the differences.
Manual intervention: When deployments fail, or environments drift, teams resort to manual fixes, creating snowflake infrastructure that becomes increasingly difficult to troubleshoot and impossible to recreate.
Limited audit trail: Traditional ops depends on a system to capture change and this lacks comprehensive logging of who changed what, when, and why. The level of bureaucracy required adds administrative burden without delivering a robust system for audit and compliance.
Reactive resolution: Teams typically discover configuration drift or deployment issues after problems manifest in production, leading to longer recovery times and increased customer impact.
Tool sprawl: Different teams often adopt different tools and practices, creating a heterogeneous landscape that’s difficult to secure, monitor, and maintain.
Security vulnerabilities: Traditional infrastructure management requires elevated access control to infrastructure and management portals, creating larger attack surfaces and compliance challenges.

The State of GitOps research found those in the pre-adoption phase had the lowest software delivery performance and were least likely to attain the benefits of GitOps.

1. Foundations

At the foundations stage, drift may be detected and teams notified, but correction often remains manual. The primary benefit is visibility, as teams can see what changed, when, and who made the change. This foundation creates the trust and transparency necessary for the more advanced practices that follow.

Practices:

Declarative desired state: Describes your target end state, making it easier to understand, change, and review.
Human readable format: Makes it easier to manage configuration and take advantage of version control tooling.
Version control: Provides the infrastructure for access control, storage, review, history, and audit trails.

Culture:

Generative culture / psychological safety: Ideas are welcome and it’s safe to experiment.
Shift infrastructure left: Infrastructure configuration is treated with the same care as application code.
Collaborative configuration management: Multiple team members collaborate to manage configuration.

2. Depth

At the depth stage, drift is detected and corrected automatically. This solidifies the practice of making all changes through version control, even in an emergency.

Practices:

Responsive code review: Keeps changes flowing smoothly, reduces merge conflicts, and encourages changes to be made the GitOps way.
Automatic pull: Improves security by having the agent call back to version control instead of exposing endpoints to a central system.
Continuous reconciliation: Keeps your system in the target state and encourages all changes to go through version control.

Culture:

GitOps for all changes: Changes are made the GitOps way, even in emergencies.
Shift from reactive to proactive: Focus shifts to earlier interventions for reliability.
Operational confidence: The team trusts the GitOps reconciliation loop.

3. Platform

At the platform stage, the organization is looking to apply GitOps practices universally and will seek a high level of standardization. Templates may be offered to accelerate onboarding, though they may lose consistency as teams evolve them independently.

Practices:

Recreatable applications: Proves you’ve captured all the infrastructure and application configuration required to spin up a new environment on demand.

Culture:

High degree of standardization: Many teams converge on good practices and apply them to all GitOps workflows.
Security baked in: GitOps includes security practices as part of the workflow, not a separate activity.
GRC needs met: Governance, risk, and compliance needs are considered and incorporated.

4. Scale

The scale stage is where organizations apply GitOps broadly across production workloads. This involves alignment across multiple providers, clouds, and locations so software teams are decoupled from hosting choices. Where templates are offered, mechanisms are in place to manage how template improvements are rolled out to all teams that use them.

Practices:

Template management: Lets template improvements be easily rolled out to all teams.
Cross-platform configuration management: Decouples teams from your infrastructure choices by providing a unified configuration.

Culture:

Non linear application cost: The organization can scale the breadth of adoption with sub-linear cost increases.
Continuous improvement culture: GitOps workflows and templates are regularly refreshed to improve them.
Center of excellence: The organization provides training and support, encouraging broad alignment on GitOps use.

Summary of maturity levels

All the cultural capabilities and practices are listed below, along with their maturity levels.

Level	Category	Facet
1. Foundations	Practice	Declarative desired state
1. Foundations	Practice	Human readable format
1. Foundations	Practice	Version control
1. Foundations	Culture	Generative culture / psychological safety
1. Foundations	Culture	Shift infrastructure left
1. Foundations	Culture	Collaborative config management
2. Depth	Practice	Responsive code review
2. Depth	Practice	Automatic pull
2. Depth	Practice	Continuous reconciliation
2. Depth	Culture	GitOps for all changes
2. Depth	Culture	Shift from reactive to proactive
2. Depth	Culture	Operational confidence
3. Platform	Practice	Recreatable applications
3. Platform	Culture	High degree of standardization
3. Platform	Culture	Security baked in
3. Platform	Culture	GRC needs met
4. Scale	Practice	Template management
4. Scale	Practice	Cross-platform configuration management
4. Scale	Culture	Non-linear cost of applications
4. Scale	Culture	Continuous improvement culture
4. Scale	Culture	Center of excellence

Adoption tips

Culture first: Starting with culture and education is how you achieve the best outcomes. DevOps research has shown that generative culture and psychological safety are the pre-requisites for high-performing teams and GitOps is no different. When culture is prioritized over tooling, organizations report better adoption outcomes.

Start small, think big: The data for breadth of GitOps adoption highlighted how organizations begin with a single application that’s representative of a large set of use cases, allowing them to polish the skills and learn tools before rolling GitOps out to many similar applications. This slow-slow-fast approach maximizes learning opportunities, reduces risk, and provides a springboard for broadening adoption quickly.

Measure continuously: The outcomes described in the research also provide useful measures of your GitOps progress. You can self-assess against the DevOps and GitOps outcomes and use what you learn to improve your adoption. Data-driven improvement accelerates maturity progression and helps you recognize and avoid bias.

Ready to improve your GitOps?

Use the GitOps self-assessment to see where you stand and get specific recommendations for improvement. You can also dive deeper into the research behind these practices in the State of GitOps report.

What is GitOps? A research-backed answer!

Mon, 13 Oct 2025 15:13:32 +0000

GitOps combines the best ideas from DevOps and infrastructure as code (IaC). By storing definitions of infrastructure and applications in version control, teams can use familiar tools to make, review, rollback, and audit those changes.

Just like DevOps, there are cultural and technical elements to GitOps. Teams agree to flow all changes through declarative state files and decide on the collaboration around changes. Once a team makes a change, high levels of automation ensure they are applied.

Let’s look at how GitOps was defined and then validate it through the State of GitOps research.

How GitOps works: The reconciliation loop

At the core of GitOps is a reconciliation loop that:

Observes the system’s current state
Compares it to the desired state
Makes adjustments to align them

The loop runs continuously, keeping your system in sync with the configuration in version control.

The 4 GitOps principles

The Open GitOps project provides 4 technology-agnostic principles that define a widely accepted definition of GitOps.

1. Declarative

A system managed by GitOps must have its desired state expressed declaratively.

You must express your system’s desired state declaratively. Instead of writing step-by-step instructions, you describe what you want the result to look like. This approach separates what you want from how to get there, eliminating problems caused by assumptions about starting conditions.

2. Versioned and Immutable

Desired state is stored in a way that enforces immutability, versioning, and retains a complete version history.

Store your desired state in version control to ensure it’s versioned, immutable, and maintains a complete history. This lets you to diagnose issues and roll back changes when needed.

3. Pulled Automatically

Software agents automatically pull the desired state declarations from the source.

Software agents automatically pull configuration from version control. This means your configuration doesn’t need to know where it’s going or how many destinations it serves.

4. Continuously Reconciled

Software agents continuously observe the actual system state and attempt to apply the desired state.

Agents continuously monitor the actual system state and work to align it with the target state defined in version control.

The 6 practices that make GitOps work

GitOps research defines 6 specific practices that drive successful GitOps outcomes. These practices work together, so you need all of them to get the full benefits, though you don’t have to implement them all at once.

Declarative desired state
Human readable format
Responsive code review
Version control
Automatic pull
Continuous reconciliation

1. Declarative desired state

Declarative configuration gives you a clear target to aim for. Unlike imperative approaches that require you to think through every step, declarative configuration describes your target end state. This makes it easier to understand, change, and review.

Benefits:

Clear documentation of your target state
Simpler changes and reviews
Readable change history
Automation handles the complexity

2. Human readable format

If humans can’t easily read your configuration files, you lose most of the value that version control provides. Instead of simple diffs, you’d have to load different versions into specialized tools for comparison.

Benefits:

Changes follow familiar developer workflows
Reviews happen within standard code review tools
Version control provides natural change history
Configuration serves as documentation

3. Responsive code review

Quick reviews keep changes flowing smoothly. Slow GitOps reviews create delays that encourage teams to work in larger batches, which is where most software delivery problems start. When reviews are slow, teams often bypass GitOps entirely, losing the benefits of review and change tracking.

Benefits:

Smaller, less risky changes
Version control becomes the primary way to change systems
Better throughput with fewer bottlenecks

4. Version control

While many consider version control fundamental to GitOps, the other practices make version control truly effective. Version control provides the infrastructure for access control, storage, review, history, and audit trails.

Benefits:

Single source of truth for system state
Familiar tools and processes
Complete change history with rollback capability

5. Automatic pull

When you use agents to pull configuration from version control improves security, you don’t need to expose endpoints for remote commands. Instead of a central system pushing changes out, agents collect configurations on their own schedules.

Benefits:

Better security with no exposed endpoints
No need to coordinate multiple destinations
Simpler architecture

6. Continuous reconciliation

Continuous reconciliation keeps your system in the target state and encourages all changes to go through version control. Manual changes get detected and corrected quickly. While you can implement this with human intervention, automatic reconciliation works better and faster.

Benefits:

Confidence that systems stay in the intended state
Strong incentive to make all changes through version control
Faster detection and correction of drift

Getting Started

Most teams shouldn’t try to implement all 6 practices at once. A gradual approach works better. Establish one practice, build your skills, and then add the next one. This gives you time to work through challenges, like creating declarative configurations for existing applications.

Ready to Assess Your GitOps Maturity?

GitOps for Enterprise Modernization: Your Roadmap from Legacy to Velocity

Mon, 22 Sep 2025 20:47:37 +0000

Many enterprises still rely on legacy infrastructure, slow manual processes, and siloed IT teams. The result is fragile systems that can’t keep up with rapid business demands. That’s why more organizations are adopting gitops to modernize their operations.

Gitops is not just a buzzword. It’s a framework that uses Git as the single source of truth for deploying and managing infrastructure and applications. Every change is version-controlled, auditable, and automated. For enterprises, it represents a clear path from legacy complexity to modern agility.

Why Legacy Approaches Fall Short

Legacy systems often involve manual updates, ticket-driven workflows, and disconnected monitoring tools. This creates:

Long deployment cycles
Higher error rates from manual changes
Limited visibility across teams
Security and compliance gaps

Research shows that nearly 40% of enterprises lack adequate internal AI and automation skills, making legacy inefficiencies even harder to overcome. In a digital-first market, these barriers can delay innovation and reduce competitiveness.

What GitOps Brings to the Enterprise

Enterprises adopting gitops report measurable improvements in speed, governance, and reliability. According to Microsoft’s 2025 Work Trend Index, 42% of companies plan to use multi-agent systems to automate complex tasks within five years, with gitops serving as the operational backbone.

Key benefits include:

Automation by default – Changes applied automatically from Git repositories reduce human error.
Auditability – Every deployment is versioned and traceable for compliance.
Scalability – Gitops scales across multi-cloud and hybrid environments, aligning with growth.
Resilience – Rollbacks are fast and reliable, protecting uptime in critical systems.

A Roadmap for Modernization with GitOps

Shifting from legacy practices to gitops requires a clear roadmap. Here’s how enterprises can approach it:

Assess the Current State
Start with an honest review of infrastructure, processes, and pain points. Most enterprises still operate with fragmented automation. A 2025 ControlMonkey survey found only 1% of IT teams had fully automated infrastructure management.
Establish Git as the Source of Truth
Move all configuration files, manifests, and policies into Git repositories. This creates a reliable audit trail and ensures teams work from a single version of truth.
Automate Deployments
Adopt tools such as ArgoCD or Flux to connect Git with your deployment pipelines. These tools automatically apply changes and monitor drift between the repository and live environments.
Build Governance into the Process
Enterprises face high stakes with compliance. Gitops enforces governance by requiring pull requests, approvals, and reviews before changes take effect. This aligns with stricter industry regulations.
Scale Across Teams and Clouds
As organizations grow, gitops scales by standardizing workflows across multiple environments. With global cloud adoption expected to grow at a 21% CAGR through 2034, gitops ensures consistency across providers like AWS, Azure, and GCP.

Real-World Example

A financial services enterprise running legacy applications struggled with compliance audits and deployment delays. By moving to gitops, they centralized all configurations into Git, automated their CI/CD pipelines with ArgoCD, and reduced deployment times by 60%. Compliance audits improved because every change was documented in Git history, making governance easier.

Challenges to Anticipate

The shift to gitops isn’t without hurdles. Common obstacles include:

Skill gaps – Many IT teams need training in Kubernetes, Terraform, and CI/CD workflows.
Cultural resistance – Moving from manual control to automation can face pushback.
Tool sprawl – Enterprises must consolidate tools to avoid adding more complexity.

Addressing these challenges requires leadership commitment and a clear upskilling plan.

Final Word from UpTech Solution

Gitops is quickly becoming the standard for enterprise modernization. It provides speed, compliance, and resilience that legacy practices cannot deliver. While challenges exist, the long-term benefits are clear: faster delivery, stronger governance, and scalable growth.

At UpTech Solution, we help enterprises adopt gitops practices that align with their business goals. By combining infrastructure automation with the right talent, we ensure modernization efforts deliver real results.

What GitOps changes about elevated access

Wed, 30 Jul 2025 03:04:39 +0000

Recently at Octopus Deploy, we surveyed the industry to gain insights into the adoption and challenges of real-world GitOps, with the results forming the State of GitOps report. While reviewing the trends and results, the data around one key finding jumped out at me:

GitOps reduces elevated access

Overall, 66% of respondents agreed. Among organizations with higher GitOps maturity, agreement rose to 77%, but interestingly, there was also a slight increase in disagreement among the highest performers.

So, does GitOps change or reduce the need for elevated environment access, and should it?

How GitOps shifts access away from infrastructure

The 4 GitOps principles encourage you to avoid manually logging into environments and performing tasks. The goal is to manage system state declaratively through version-controlled configuration and apply those states automatically using reconciliation agents.

If fully embraced, GitOps agents continuously observe the system state, correcting any drift by reapplying the desired state defined in version control. Continuous reconciliation discourages manual changes and automatically reverses them if they occur.

In traditional operations teams, access to systems with SSH or tools like kubectl is common, but operating in a mature GitOps model should make these break-glass exceptions.

Most survey respondents agree that GitOps should reduce the need for privileged production access by replacing hands-on changes with declarative, auditable workflows.

Rethinking what elevated access means with GitOps

So why might some respondents to the survey disagree that embracing GitOps reduces elevated access?

At lower maturity levels, teams may treat Git as a safe, developer-focused system. But as Git becomes the source of truth for managing production infrastructure and applications, a key question emerges:

If a developer can change a manifest in Git for a production environment, does that emulate elevated access?

In a GitOps model, Git becomes the control plane, meaning write access to a production configuration in Git, especially with automatic reconciliation, becomes a form of production access.

High performers likely consider GitOps secure when configuring repositories well and implementing mature environment promotion workflows. However, Git isn’t the only sensitive surface in GitOps architecture. You must also secure the reconciliation controller and supporting workflow or promotion tooling with the same level of rigor. Together, these systems form the modern delivery pipeline and deserve the same access scrutiny once reserved for direct infrastructure.

High performers likely aren’t ignoring or rejecting GitOps’ security benefits, but instead acknowledge that you must treat Git itself as a sensitive operational boundary now more than ever. It’s now one of several critical control points in the delivery process that need strict governance.

Interestingly, these same high performers agreed that GitOps reduces overall elevated access. So, the takeaway isn’t contradiction; it’s maturity. High-performing teams see GitOps as secure only when Git, the reconciliation controller, and supporting environment progression tooling are all treated as critical access points, with the appropriate controls in place.

Handling exceptions to the GitOps model

In the early stages of adopting GitOps, it’s common for DevOps engineers, platform engineers, and even developers to rely on familiar tools like kubectl to inspect or tweak clusters and applications directly. Especially in learning environments and environments early in the deployment process, like dev, this is precisely what they’ll do. This approach lets them rely on familiar tools while gaining confidence in GitOps practices.

The ultimate goal is for nobody to have direct access to production environments, even if read-only.

In the State of GitOps report, we surfaced six core practices of GitOps:

Declarative desired state
Human-readable format
Responsive code review
Version control
Automatic pull
Continuous reconciliation

You won’t embrace all of these GitOps practices on day one, and that’s ok. As we learned from the DevOps movement, embracing a GitOps infrastructure and application delivery approach involves continuous improvement, learning, tweaking, and time.

With that said, even at high maturity levels with GitOps where you implement many or all of the practices, there may still be valid scenarios where folks need elevated access. For example, emergency responses (break-glass scenarios) or debugging complex or unreproducible issues.

In these cases, it’s crucial to have:

Break-glass procedures. Documented steps for requesting, granting, and revoking temporary access. These should include automated expiration, require multi-party approval, and be used only in exceptional circumstances.
Audit trails. You must log every access event, recording who accessed and changed what, when, and why.
Controlled suspension of reconciliation. Sometimes, you must temporarily disable the GitOps agent to prevent it from overwriting a manual change made during incident response. You should log and time-bound this suspension, and include procedures to reconcile safely afterward.
Post-incident recovery. After manual intervention, have a defined process for reconciling the system back to the desired state, committing required changes to Git, and re-enabling automatic reconciliation.

These situations should be rare. Ideally, releases have already passed through multiple environments that closely mirror production in both infrastructure and configuration. If you experience an issue where you find your developers or engineers need to access environments directly, after the issue gets resolved, question why. Could they have achieved the same outcome by using other systems and tooling? Is there an element of education and training required? Is there a need to implement supplementary tooling or processes to help troubleshoot and resolve issues that surface after deployment? Work on fixing it, and implementing that fix in your process so it doesn’t occur again.

Releasing to production shouldn’t be a production.

The evolving access conversation

GitOps won’t immediately eliminate the need for you to have a process for elevated production access, but it should help make it an exception rather than a routine operation. You must treat Git and GitOps tooling with the same care and control as production infrastructure. The fundamentals still apply; they apply in more places and different places than you’re used to.

As you mature your GitOps adoption, the conversation shifts from “do we still need access?” to “are we securing the new access surface and tooling?”.

GitOps Makes for Great DevEx, but Pragmatism Matters

Mon, 21 Jul 2025 11:52:53 +0000

At Kubecon Paris, we surveyed expert developers about their modernization initiatives, including what works well and where there’s pain.

KubeCon in Paris was an amazing opportunity for like-minded people to gather together to talk about Kubernetes, GitOps and cloud native computing. We took the opportunity to survey expert developers about their modernization initiatives, including what works well and where there’s pain.

Developer experience (DevEx) is the primary driver for modernization initiatives. More than two-thirds of organizations are moving to cloud native, Kubernetes and GitOps to improve DevEx. Despite these good intentions, there are two common trip hazards that must be addressed:

Developers want pragmatic GitOps, not pure GitOps.
Developers need better CD tools, not just a tool that runs custom scripts.

Developers Want Pragmatic GitOps

GitOps and Kubernetes are natural companions. Compared to virtual machines, Kubernetes has superpowers in container orchestration and dynamic scaling. The ability to automate Kubernetes is also a huge advantage, and GitOps is the natural way to do this.

Pragmatic GitOps stops short of saying everything should happen in version control. For example, a dedicated CD tool will handle your Kubernetes manifest file and handle the deployment process. It will apply the correct configuration for each environment and ensure software versions progress through each environment.

The problem with the pure-GitOps approach is that you end up with YAML files everywhere. Not just two or three files, but thousands. It only takes a few services and environments to generate YAML sprawl, as illustrated below. If you apply different configurations per customer or location, the problem is even worse.

We spoke to Kubernetes experts at KubeCon Paris. We found twice as many advocated for pragmatic GitOps over a pure approach. (Pragmatic GitOps was preferred by 67% of experts vs. 33% who opted for pure GitOps.) As DevOps teams apply GitOps at scale, the need for specialized tooling to handle things like configuration becomes more apparent.

Hammers are great at what they do, but they don’t do everything. Once you’ve experienced YAML sprawl, you’ll double-check your toolbox to see what else you have available for the job.

Developers Want Better Continuous Delivery

We know that DevEx is the №1 driver for modernization initiatives, but developers want better continuous delivery tools.

The current problem stems from the fact that too many teams are using their CI tools to run CD tasks (but CI is not CD). This means they are writing scripts they then store in strings inside of YAML files. CI tools don’t offer features for environment progression, deployment observability and configuration management for environments and tenants.

While these hand-cranked scripts run inside a CI tool, they are actually more like a homegrown scripted deployment solution.

You have likely encountered a shadow IT project at some point in your career, where a business user has created a critical application inside an Excel spreadsheet. These inevitably become urgent rescue projects when their creator moves on or becomes overwhelmed with what they created. Creating a CD solution inside a CI tool is basically do-it-yourself shadow CD.

Almost two-thirds of developers are using a CI tool or all-in-one solution for their CD pipeline, and it’s leading to an explosion in self-managed scripts as a basis for deployments. This is one of the reasons DevEx is a problem to be solved. More than 80% of developers we surveyed told us that CD is the most impactful investment that can be made for DevEx.

As organizations invest in improving DevEx and software delivery performance, an appropriate consideration of CD will be crucial.

Update Good Practices for Success

It has long been held that modern software development is fraught with cognitive overload. When you first pick up Kubernetes, the ecosystem of choices causes decision overload. Once you’re up and running, though, it’s YAML sprawl and DIY shadow CD that are limiting your software delivery and causing pain for developers.

In the “Leader’s Guide to Kubernetes Adoption,” Nikita Dergilev and Bob Walker explain that business goals need to drive your Kubernetes adoption to avoid being deprioritized by future initiatives. There are a number of decisions to make along the way that will benefit from a collaborative standardization process. The two crucial choices you can make are with respect to DIY shadow CD and YAML sprawl.

We’ve moved on from traditional infrastructure because modern cloud native solutions offer so many benefits. But unintentionally, we’ve lost finesse in the way we manage software versions. Updating our good practices to support modern software is critical if we want to solve the problems of cognitive load.

GitOps vs Traditional CI/CD: A Comprehensive Comparison for Modern DevOps Practices

Fri, 18 Apr 2025 06:40:25 +0000

Introduction

The world of software development has seen immense changes over the last decade, especially with the growing adoption of cloud-native applications and microservices. Two of the most widely used concepts in modern DevOps practices are **GitOps** and **Traditional CI/CD**. While both aim to automate the deployment and management of applications, they differ in fundamental ways. In this article, we will compare GitOps with traditional Continuous Integration/Continuous Deployment (CI/CD) processes, explain their unique characteristics, and highlight how they fit into modern software development workflows.

As DevOps teams continue to seek more efficient ways to handle application deployment and infrastructure management, it’s essential to explore the benefits and challenges of both GitOps and traditional CI/CD. This comparison will provide clarity on how each method works and how they impact the overall software delivery pipeline, focusing on areas like automation, security, scalability, and ease of use. Additionally, we will dive into real-world examples to demonstrate how each model can be effectively implemented in various environments.

What is GitOps?

GitOps is a practice that revolves around using **Git repositories** as the source of truth for both infrastructure and application deployment. The core idea behind GitOps is that every change to the infrastructure and application configuration is made through a commit to a Git repository, allowing Git to become the central point of reference for both code and infrastructure. This practice greatly simplifies the deployment process and enhances collaboration among DevOps teams, providing a clear audit trail for all changes made to the system.

The concept of GitOps is deeply tied to **Infrastructure as Code (IaC)**. IaC is the practice of managing and provisioning computing infrastructure through code rather than manual processes. In GitOps, everything about the application and its environment is described declaratively in version-controlled files, most commonly YAML or JSON. These configuration files describe how the system should look and behave, allowing GitOps tools like **ArgoCD** and **Flux** to continuously monitor and update the state of the system to match the desired state described in Git.

GitOps enables **pull-based deployment**. This means that rather than pushing configuration changes directly to production environments, GitOps tools monitor Git repositories for changes and pull them into the live environment, applying updates as needed. If discrepancies occur between the desired state (as described in Git) and the actual state in the environment, GitOps tools automatically correct the situation, ensuring consistency across environments.

This approach provides several advantages over traditional deployment methods. By using Git as the source of truth, organizations gain better **visibility**, **auditability**, and **reproducibility** of their infrastructure and deployments. It also supports more seamless **collaboration** among team members, as any change to the environment can be traced back to a specific commit in the Git repository.

What is Traditional CI/CD?

**Continuous Integration (CI)** and **Continuous Deployment (CD)** are foundational practices in modern software development. They aim to automate various stages of the software lifecycle, from building and testing to deploying applications in production. While CI/CD systems have been widely used for years, they function differently from GitOps.

**Continuous Integration** focuses on automating the process of integrating new code changes into a shared code repository. When a developer commits code to the repository, the CI pipeline triggers automated processes to build and test the code. This ensures that new code is integrated smoothly, without introducing bugs or errors. CI is primarily focused on code quality and efficiency in the development cycle.

**Continuous Deployment**, on the other hand, focuses on automating the deployment process. Once the CI pipeline has verified the new code, the CD pipeline takes over and deploys the code to staging or production environments. Tools like Jenkins, GitLab CI, and CircleCI are typically used to automate these processes, which can include running tests, building artifacts, and managing deployment pipelines.

Unlike GitOps, traditional CI/CD systems are generally **imperative**. In an imperative approach, a set of predefined commands and steps are executed to perform the deployment. These steps are often specified in configuration files or scripts, making the process more flexible but also more prone to errors if not managed carefully.

While CI/CD systems can offer robust automation for building and deploying applications, they typically rely on multiple tools for managing different parts of the pipeline. For example, the CI tool handles the build and test stages, while the CD tool handles deployment. This often results in a fragmented deployment process, which can introduce complexity in managing and monitoring deployments across multiple environments.

GitOps vs Traditional CI/CD: Key Differences

1. Source of Truth

The primary difference between GitOps and traditional CI/CD lies in the **source of truth**. In GitOps, the source of truth is a Git repository. All configuration files, including infrastructure, application deployment, and environment settings, are stored and versioned in Git. This makes Git the central point of reference for both development and operations teams. By using Git as the source of truth, GitOps ensures that the infrastructure and applications are always in sync with what is declared in the repository.

In traditional CI/CD systems, however, the source of truth is distributed across various tools and systems. For example, the application code might be stored in a Git repository, but deployment configurations, infrastructure as code, and other system settings could be managed using separate tools like Terraform or Ansible. This decentralized approach can make it difficult to keep track of all the components of a system and ensure that they are consistently deployed.

2. Workflow Automation

GitOps introduces **highly automated workflows**. Since the Git repository acts as the central point of truth, changes to infrastructure and applications are automatically detected and applied by GitOps tools. For example, when a change is committed to the repository, tools like ArgoCD or Flux automatically reconcile the live environment to match the desired state described in the Git repository.

In contrast, traditional CI/CD workflows often require more manual intervention. Although CI pipelines automate code building and testing, CD pipelines may require manual approval steps for deployments, especially to production environments. This can create bottlenecks in the deployment process, especially when you need to quickly roll out fixes or updates.

3. Rollback and Auditing

GitOps has a **native rollback mechanism**. If an issue arises in the deployed application or infrastructure, a rollback can be as simple as reverting a commit in the Git repository. This provides a clear audit trail of changes, making it easier to trace when and why a specific change was made. The use of Git as the source of truth provides a secure, version-controlled history of changes, which can be invaluable for troubleshooting and compliance auditing.

Traditional CI/CD systems can be more complex in terms of rollback. While most tools offer rollback features, they are typically more manual and require explicit configuration. In some cases, it may require rolling back multiple layers of the application or environment, which can introduce inconsistency if not managed carefully.

4. Declarative vs Imperative

GitOps operates in a **declarative** model. The desired state of the system is defined in Git, and the system automatically reconciles to match that state. This is much simpler because the Git repository holds a **single source of truth** that describes how the system should behave.

In traditional CI/CD, most systems are **imperative**, meaning they rely on step-by-step commands to execute specific actions in the pipeline. These pipelines are more customizable and flexible but may require more management and scripting to handle complex deployment scenarios.

5. Monitoring and Observability

GitOps enhances **monitoring and observability** by providing a single point of reference. Since everything is described in Git, it's easier to track changes, view the state of the system, and monitor any discrepancies between the desired and actual state. GitOps tools continuously check and apply the desired state, giving teams greater visibility into the deployment process.

Traditional CI/CD systems also offer monitoring capabilities, but these are often fragmented across different tools. For example, Jenkins might provide logs about the build process, while Kubernetes might provide monitoring data on containers and pods. GitOps provides a unified view, making it easier to see the full picture in real-time.

Benefits of GitOps Over Traditional CI/CD

GitOps offers several significant advantages over traditional CI/CD pipelines, particularly in environments that rely heavily on cloud-native applications and containerized systems like **Kubernetes**. These benefits make GitOps an attractive option for organizations looking to modernize their DevOps workflows.

1. Improved Automation and Consistency

GitOps takes automation to the next level. The fact that changes to infrastructure and application code are made through Git commits ensures that the system is always in sync with the latest changes. GitOps tools like **ArgoCD** and **Flux** continuously monitor the repository and automatically apply any changes to the live environment. This eliminates the need for manual intervention and ensures consistent environments across all stages of development.

2. Enhanced Security and Auditing

By using Git as the source of truth, GitOps provides **better security** and **auditing** capabilities. Every change made to the environment is version-controlled and auditable. This creates a clear history of who made which changes and when, which is crucial for tracking down issues, ensuring compliance, and maintaining overall system integrity. Unauthorized changes can be easily detected, and the rollback process is as simple as reverting a Git commit.

3. Simplified Rollbacks

GitOps makes **rollback operations** easier and more reliable. Since the desired state of the system is stored in Git, rolling back to a previous version is simply a matter of reverting to a prior commit. This is far simpler than traditional CI/CD systems, where rollback may require manually undoing multiple steps in the pipeline and ensuring that all dependencies are handled correctly.

4. Scalability and Flexibility

GitOps is particularly well-suited to managing large, complex systems, especially in **microservices** and **Kubernetes** environments. Because the configuration is declarative, it’s easier to scale and manage numerous microservices or infrastructure components. GitOps tools can monitor and deploy changes to thousands of services with minimal manual intervention, making it ideal for large organizations with diverse infrastructure needs.

Challenges and Limitations of GitOps

While GitOps offers numerous advantages, it is not without its challenges. For example, the initial setup of a GitOps workflow can be complex and require significant effort to integrate with existing systems. Additionally, GitOps works best in environments that are entirely cloud-native or use Kubernetes, so organizations that rely on traditional infrastructure may face difficulties adopting GitOps.

Another limitation is that GitOps introduces a level of abstraction, which can make troubleshooting more difficult in some cases. While GitOps provides clear audit trails, issues with the automation tools themselves or with misconfigurations in the Git repository can cause disruptions, requiring teams to dig deeper into the underlying tools and systems.

Despite these challenges, GitOps is rapidly becoming a go-to approach for many organizations looking to modernize their DevOps pipelines and embrace the benefits of **automation**, **consistency**, and **scalability**.

Conclusion

GitOps and traditional CI/CD systems both have their strengths and weaknesses, but GitOps offers significant advantages when it comes to automation, consistency, and security. The ability to use Git as the source of truth for both code and infrastructure is a game-changer for teams looking to streamline their deployment processes, reduce human error, and improve collaboration.

For organizations already using Kubernetes or other cloud-native technologies, GitOps presents a natural fit, allowing them to automate their deployments and infrastructure management seamlessly. Traditional CI/CD systems, while still powerful and widely used, may require more manual steps and complex configurations. Ultimately, the choice between GitOps and traditional CI/CD depends on the organization's specific needs and the complexity of its infrastructure, but GitOps is undeniably a modern approach to DevOps.

Coder Legion - Recent posts tagged gitops

The GitOps Bridge: Keep Terraform and ArgoCD Out of Each Other's Way

🌉 The GitOps Bridge: Keep Terraform and ArgoCD Out of Each Other's Way

The coupling you do not notice until it hurts

The contract

Why this made a hard migration tractable

How to adopt it

The reality of GitOps application recreation

The real-world value of recreatable applications

The maturity journey

What to consider with complete recreation

The path forward

Beyond InferenceService Readiness: 5 GitOps Failure Modes That Break KServe Deployments

The platform context

Failure Mode 1: Webhook Has No Endpoints — Sync Fails Cluster-Wide

Symptom

Root Cause

Fix

Lesson

Failure Mode 2: CRD Deleted by a Misapplied Patch — All Endpoints Gone Instantly

Symptom

Root Cause

Fix

Lesson

Failure Mode 3: Permanent OutOfSync Due to Label Key Mismatch

Symptom

Root Cause

Fix

Lesson

Failure Mode 4: Kyverno Install Breaks ArgoCD Reconciliation Loop

Symptom

Root Cause

Fix

Lesson

Failure Mode 5: Stale Admission Webhook Blocks All Resource Creation

Symptom

Root Cause

Fix

Lesson

The Triage Sequence That Saves Hours

Why This Matters for Platform Teams

What I Would Improve Next

See Also

ArgoCon Lightning Keynote: Argo CD vs the world

6 GitOps Practices That Actually Work

1. Declarative Desired State

2. Human Readable Format Is Critical

3. Responsive Code Review

4. Version Control

5. Automatic Pull

6. Continuous Reconciliation

It’s Not Gatekeeping, It’s Data

Autopilot - the Final: From Localhost to the Cloud with GitHub Actions and CI/CD

The Pipeline Architecture

The Orchestrator (orchestrator.py)

The Star Feature: "Director's Cut"

Security and CI/CD: Sleeping Soundly

The Final Result

Autopilot Project Conclusion

SHIPPED25: The Research Panel

GitOps Gap: Few Use Declarative Configuration To Manage State

What Is Declarative Desired State?

Why Declarative Desired State Is Crucial

Not a Simple Task

Roles and Regions Are on a Journey

Declarative Configuration Will Rise

The GitOps maturity model

The statistical model

Pre-adoption

1. Foundations

2. Depth

3. Platform

4. Scale

Summary of maturity levels

Adoption tips

Ready to improve your GitOps?

What is GitOps? A research-backed answer!

How GitOps works: The reconciliation loop

The 4 GitOps principles

1. Declarative

The Orchestrator (`orchestrator.py`)