Coder Legion - Recent posts tagged ingress

Same Ingress, Different Defaults: What Changed When We Moved to HAProxy

Thu, 08 Oct 2026 08:23:47 +0000

Over the last few weeks, we moved the customer-facing managed services on our NKE service clusters and Deploio from ingress-nginx to HAProxy: Grafana, Loki, ArgoCD, the container registry, Alertmanager, Tempo, OpenSearch, the Object Storage endpoints and all Deploio applications. Some clusters alone had more than 200 Ingresses.

We have written about the motivation twice before. In December, we explained what the retirement of ingress-nginx meant and which migration paths were open. Our TechTalk #29 recap covered our evaluation of Traefik, Gateway API and HAProxy, and why HAProxy became our default for classic Ingress on NKE.

This post is about what happened after that decision.

The annotation mapping table in our documentation gets you a working Ingress on HAProxy in a few minutes. What it does not show is how the defaults shift underneath your workloads. Two ingress controllers can parse the same Ingress resource and still differ in how they talk to your backends, which status codes they send, how long they wait and how many connections they accept before refusing new ones.

We found most of these differences in production, after the switch. One of them broke a customer’s website asset delivery. We are sharing them so you can check your own setup before you migrate.

HTTP to HTTPS Redirects

While comparing responses from both controllers, we noticed that plain HTTP requests were redirected with 302 Found on HAProxy, whereas nginx returned 308 Permanent Redirect.

ingress-nginx sets http-redirect-code to 308 by default. haproxy-ingress defaults its corresponding ssl-redirect-code to a temporary 302. Browsers follow both, but API clients, search crawlers and intermediate caches treat them differently. A 302 allows clients to rewrite a POST into a GET on redirect, whereas a 308 preserves the request method and body.

A silent downgrade to temporary redirects can break API consumers, so we set ssl-redirect-code: "308" globally on our managed controllers to keep the previous behaviour.

Connection Limits

During peak traffic, requests to Deploio applications slowed down and some new connections were refused.

The cause was the connection limit. nginx and HAProxy calculate their ceilings differently:

nginx multiplies worker_processes by worker_connections. With
ingress-nginx setting worker_processes to the node’s CPU count and
worker_connections to 16384, even a modest node can handle tens of
thousands of concurrent connections.
HAProxy enforces a single global maxconn, which haproxy-ingress
exposes as max-connections and sets to 2,000 per pod by default.

Once a pod reaches that ceiling, HAProxy stops accepting new connections. They then queue in the kernel’s accept backlog, which caused the slowdown. Once the backlog is full, they are dropped.

We raised max-connections to 20,000 across all managed controllers.

Timeouts

Shortly after we migrated Deploio, some customers saw requests fail with 504 Gateway Timeout.

haproxy-ingress defaults timeout-client and timeout-server to 50 seconds, while ingress-nginx waits 60 seconds. Ten seconds make no difference for typical web requests, but data exports or reports that reliably finished in 55 seconds suddenly failed with a 504.

To avoid cutting off requests that used to succeed, our managed controllers set both timeout-client and timeout-server to 60 seconds. You can still override the timeout per Ingress with haproxy-ingress.github.io/timeout-server in place of nginx.ingress.kubernetes.io/proxy-read-timeout.

Upstream HTTP Version

In July, we moved an S3-compatible Object Storage endpoint to HAProxy. A customer served website assets through their own nginx reverse proxy in front of that endpoint. Right after the cutover, every asset request failed with 400 Bad Request.

The cause: nginx’s proxy_pass talks to upstreams over HTTP/1.0 by default. ingress-nginx always spoke HTTP/1.1 to backends, regardless of the client’s protocol, and so it silently upgraded the customer’s HTTP/1.0 requests. HAProxy forwards the request with the protocol version the client sent. The Object Storage backend rejected HTTP/1.0 requests, which produced the 400 errors.

The fix was for the customer to add proxy_http_version 1.1 to their reverse proxy configuration:

location / {
    proxy_pass http://storage-endpoint;
    proxy_http_version 1.1;
}

If you run a reverse proxy in front of an NKE or Deploio Ingress, check which HTTP version it sends.

Basic Auth

After we had migrated most managed services, one of the controllers spiked to more than two full CPU cores while serving only 155 concurrent connections. Almost all traffic to that pod hit two endpoints protected by HTTP basic authentication, one of them our metrics ingestion pipeline.

HAProxy verifies basic authentication via crypt(3) synchronously on every request, without caching. A bcrypt hash with a cost factor of 10, the default in most standard libraries and in Helm’s htpasswd template function, takes about 50 ms of CPU time to verify. At 50 ms per check, twenty requests per second are enough to saturate a whole CPU core.

This also applies to nginx, as its auth_basic module verifies every request synchronously and caches nothing either. Because each nginx worker is a single-threaded event loop, a 50 ms verification stalls every connection on that worker. But workers are separate processes, so a controller with eight workers can verify eight passwords in parallel.

HAProxy uses the thread-safe crypt_r(3) only on glibc and FreeBSD. On every other platform, including the Alpine-based image that haproxy-ingress ships, it wraps the plain crypt(3) call in a single process-wide spinlock. Only one thread can verify a password at a time, which caps the whole pod at roughly twenty verifications per second, no matter how many cores it has. And because it is a spinlock, waiting threads do not sleep. They keep their cores busy until it is their turn. That is how a pod serving 155 connections ended up using more than two cores.

The HAProxy manual warns that hashed passwords “can quickly become a major factor in HAProxy’s overall CPU consumption” and notes that musl’s hash implementations are slower than glibc’s. An upstream request to cache successful basic authentication results has been open since 2019 in haproxy/haproxy#370.

For credentials we generate internally, we reduced the bcrypt cost to 4, where verification takes under one millisecond. This is safe because those passwords are long, machine-generated strings of 20 random characters, which makes brute-force attacks computationally infeasible regardless of the cost factor. Helm’s built-in htpasswd function always hashes with bcrypt’s default cost of 10 and offers no way to change it (Sprig source), so chart-generated credentials need to be hashed outside Helm.

The annotation syntax is simpler on HAProxy. Setting haproxy-ingress.github.io/auth-secret alone enables authentication, auth-type is deprecated. The secret format stays the same: an htpasswd file stored under the auth key.

X-Forwarded-For

On paper, this looks like a change in behaviour. In practice it is not one, but it does add a header.

ingress-nginx replaced any incoming X-Forwarded-For with the connecting IP unless the cluster-wide compute-full-forwarded-for option was enabled, and on our shared controllers it was off.

HAProxy’s option forwardfor on its own behaves differently: it appends the connecting IP, so if the client already sent an X-Forwarded-For, the backend receives both. Applications behind a CDN or reverse proxy would then see the whole proxy chain instead of the single value ingress-nginx gave them.

haproxy-ingress closes that gap. Its forwardfor key defaults to add, which deletes any incoming X-Forwarded-For before HAProxy inserts the connecting IP. Your application therefore sees the same as before: a single address, the one that connected directly to the ingress, for example your CDN.

What is new is one extra header. If the request already carried an X-Forwarded-For, haproxy-ingress copies its original value to X-Original-Forwarded-For before discarding it:

X-Forwarded-For: <cdn-or-proxy-ip>
X-Original-Forwarded-For: <whatever-the-client-sent>

On its own, X-Original-Forwarded-For is untrusted input. It is copied verbatim from whoever connected to the ingress, and anyone can put anything into it.

It becomes useful once your application checks who connected. X-Forwarded-For holds the address that opened the TCP connection to HAProxy, which the client cannot forge. If that address belongs to a CDN you trust, for example one of the published edge IPs of your provider, then X-Original-Forwarded-For is the header that CDN sent. Its rightmost entry is the address the CDN saw connecting to it.

HSTS Headers

haproxy-ingress defaults its Strict-Transport-Security header to max-age=15768000 (six months). ingress-nginx sent max-age=31536000 (one year).

RFC 6797 leaves the duration to the operator, and the two projects measure against different yardsticks. Six months is the minimum Qualys SSL Labs has required for an A+ grade since January 2014. HAProxy’s own documentation examples use a value just above it, max-age=16000000, described as “a bit more than 6 months”. When HAProxy published those examples, the HSTS preload list accepted a max-age shorter than one year. It has since raised the minimum to one year.

For standard web traffic, this has no functional impact. If your domain is on the HSTS preload list, however, six months fails validation. We therefore override the haproxy-ingress default and send max-age=31536000 (one year), the same value ingress-nginx did.

If you need a different duration, set spec.forProvider.hsts.maxAge on the IngressHAProxy resource (a duration, for example 8760h). For a single Ingress, set haproxy-ingress.github.io/hsts-max-age (in seconds).

Request Buffering

Under ingress-nginx, high-volume write endpoints produced warnings like this in the controller logs:

[warn] a client request body is buffered to a temporary file /tmp/nginx/client-body/0000000123
  request: "POST /loki/api/v1/push HTTP/1.1"

A while back, one node in our service cluster ran at 98% CPU, with its nginx pod using 1.4 cores mainly to write and clean up temporary disk buffers. On nginx, the fix was to set nginx.ingress.kubernetes.io/proxy-request-buffering: "off" on write-heavy Ingresses such as Loki, Promscale and container registries.

HAProxy streams request bodies to backends by default. There is no equivalent buffering annotation, because HAProxy does not write incoming payloads to disk.

Where This Leaves You

If you still use our managed ingress-nginx on NKE: as announced, support ended on 1 October 2026. HAProxy is available for every NKE cluster, and our migration documentation is the place to start.

If you are unsure how one of these differences affects your workloads, write to us at Emails are not allowed and we will look at it with you.

After NGINX Ingress Controller: Alternatives and Migration Guide

Thu, 23 Apr 2026 22:56:21 +0000

If you manage Kubernetes clusters in production, the last 18 months have been uncomfortable. Two of the most widely deployed NGINX-based Ingress Controllers have faced critical security vulnerabilities, deprecation announcements, and shifting maintenance responsibilities — all while the Kubernetes project accelerates its push toward a new traffic management standard. This is not a drill. Teams running ingress-nginx or the F5/NGINX Ingress Controller need a clear picture of what changed, what it means for their clusters, and what their realistic options are going forward.

First, Clear the Confusion: There Are Two NGINX Ingress Controllers

One of the most persistent sources of confusion in the Kubernetes networking space is that there are two completely different projects both called “NGINX Ingress Controller,” maintained by different organizations, with different architectures and different licensing.

ingress-nginx (kubernetes/ingress-nginx)

This is the community-maintained controller under the Kubernetes project umbrella, hosted at github.com/kubernetes/ingress-nginx. It uses the open-source NGINX as its data plane, configured via Lua scripting and dynamically generated nginx.conf files. This is the controller most teams end up with when they follow the official Kubernetes documentation or install from the Helm chart referenced in the ingress guide. It is free, open-source, and until recently was considered the default choice.

NGINX Ingress Controller (nginxinc/kubernetes-ingress)

This is the commercial and open-source controller maintained by F5/NGINX, hosted at github.com/nginxinc/kubernetes-ingress. It also supports NGINX Plus (the commercial version with enhanced features like active health checks, JWT authentication, and advanced load balancing). The architecture is different — it uses native NGINX APIs rather than the Lua-heavy approach — and it targets enterprise customers looking for support contracts and advanced capabilities.

These two controllers are not interchangeable. Configuration annotations differ, Helm chart values differ, and behavior under edge cases differs substantially. Understanding which one your cluster runs is the necessary starting point for any decision about migration.

# Check which NGINX IC you are actually running
kubectl get pods -n ingress-nginx -o jsonpath='{.items[*].spec.containers[*].image}'

# Community controller image looks like:
# registry.k8s.io/ingress-nginx/controller:v1.x.x

# F5/NGINX controller image looks like:
# nginx/nginx-ingress:x.x.x  or  private-registry.nginx.com/nginx-ic/nginx-plus-ingress:x.x.x

What Actually Happened: A Timeline of Disruption

The ingress-nginx CVEs (2024)

In March 2024, security researchers disclosed a set of critical vulnerabilities in ingress-nginx under the collective name IngressNightmare (CVE-2025-1097, CVE-2025-1098, CVE-2025-1974, CVE-2025-24514). The most severe of these, rated CVSS 9.8, allowed unauthenticated remote code execution against the ingress-nginx admission webhook. An attacker with network access to the admission controller could craft a malicious Ingress object to inject arbitrary NGINX configuration, ultimately achieving code execution in the controller pod — which in many clusters runs with elevated permissions and access to service account tokens across namespaces.

The vulnerabilities affected the vast majority of ingress-nginx deployments in the wild. Wiz Research, which discovered and disclosed the issues, estimated that approximately 43% of cloud environments were exposed. Patches were released in versions 1.11.5 and 1.12.1, but the incident forced uncomfortable questions about the controller’s security posture and the architecture decisions (particularly the admission webhook design) that made it possible.

Maintenance Concerns in ingress-nginx

Beyond the CVEs, the ingress-nginx project has faced ongoing concerns about maintainer bandwidth. The project is maintained by a small group of volunteers and relies heavily on community contributions. Issue response times slowed, pull requests aged, and the pace of feature development declined relative to alternatives. For a component as critical as the cluster ingress layer, this created legitimate concern about long-term sustainability without corporate backing or broader contributor growth.

F5/NGINX Deprecation Announcement

On the commercial side, F5/NGINX announced in early 2025 that the nginxinc/kubernetes-ingress controller — particularly its open-source tier — would undergo significant changes. F5 signaled a strategic shift toward NGINX Gateway Fabric, their implementation of the Kubernetes Gateway API specification. The message was clear: investment in the Ingress-based controller would be reduced, and customers were encouraged to plan migrations toward Gateway API-native solutions.

For teams running NGINX Plus-based ingress with support contracts, this was a significant business concern. The product they had licensed and standardized on was being steered toward end-of-life on the Ingress API, even if exact timelines remained somewhat ambiguous in the initial announcements.

Real Impact on Production Clusters

The practical consequences depend heavily on which controller you run and how your clusters are configured. Here is an honest assessment:

Immediate Security Risk

If you run ingress-nginx and have not patched to 1.11.5+ or 1.12.1+, your admission webhook is a critical attack surface. Patching is non-negotiable and should have happened already. The admission webhook can be disabled if you are not using it for validation (many teams are not), which significantly reduces the attack surface while you plan a longer-term migration.

# Check your current ingress-nginx version
kubectl get deployment ingress-nginx-controller -n ingress-nginx \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

# Verify admission webhook is configured
kubectl get validatingwebhookconfigurations | grep ingress

# If you need to disable the webhook temporarily (reduces but does not eliminate risk):
kubectl delete validatingwebhookconfiguration ingress-nginx-admission

Operational Uncertainty

Even after patching, the underlying questions remain. Teams are now asking: should we invest in hardening and tuning ingress-nginx knowing it may not be the strategic direction? Should we migrate now when it is our choice, rather than later when it may be forced? For NGINX IC customers, they are evaluating whether their licensing costs justify continued investment in a product being steered toward deprecation.

Configuration Migration Complexity

The real cost of migration is in the annotation-heavy configurations that accumulate over time. Teams that have built complex routing logic using nginx.ingress.kubernetes.io/* annotations — custom headers, rate limiting, auth snippets, rewrite rules, canary traffic splitting — face significant rework when switching controllers. This is the primary reason many teams are reluctant to move despite clear signals that a transition is coming.

The Alternatives: An Honest Evaluation

There is no shortage of Ingress controller options. The question is which alternatives are mature enough for production workloads at scale, and what trade-offs each brings.

Traefik

Traefik Proxy (and its Kubernetes-native version via Traefik Hub) has emerged as the most popular alternative for teams leaving ingress-nginx. It supports the standard Kubernetes Ingress API for drop-in compatibility, its own IngressRoute CRDs for advanced features, and Kubernetes Gateway API. It is written in Go, has strong TLS automation via Let’s Encrypt, and has excellent observability with built-in metrics and a real-time dashboard.

Trade-offs: Traefik’s configuration model is different enough from NGINX that complex routing logic requires rethinking rather than translating. Performance under very high connection counts is generally good but NGINX has a longer track record in extreme-scale deployments. The commercial offering (Traefik Hub) adds API gateway capabilities but introduces vendor dependency.

Envoy Gateway

Envoy Gateway is now a CNCF project and implements the Kubernetes Gateway API natively using Envoy as its data plane. This is arguably the most strategically aligned option for teams that want to bet on the future of Kubernetes networking. Envoy is battle-tested (it powers Istio, Contour, and large-scale service meshes at companies like Lyft and Google), and the Gateway API implementation is comprehensive and actively developed.

Trade-offs: Envoy Gateway is relatively young as a standalone project. Teams unfamiliar with Envoy will face a steeper learning curve for debugging and custom configuration. The operational model differs significantly from NGINX-based controllers. However, for greenfield deployments or teams willing to invest in the transition, this is a strong forward-looking choice.

Cilium Gateway API

If your cluster already runs Cilium as the CNI, enabling Gateway API support is a natural evolution. Cilium’s Gateway API implementation leverages eBPF for high-performance packet processing, avoiding the overhead of userspace proxy hops entirely. It is deeply integrated with Cilium’s network policy model and observability stack (Hubble).

Trade-offs: This option is only relevant if you are already committed to Cilium as your CNI, or are willing to make that switch simultaneously. Migrating both the CNI and the ingress layer at the same time is a significant operational risk. For Cilium shops, however, this consolidates complexity and provides excellent performance and observability.

HAProxy Ingress

HAProxy Ingress Controller is maintained by the HAProxy Technologies team and has a strong reputation for raw performance and precise traffic control. It supports both Ingress and Gateway API and has a long track record in high-throughput production environments. For teams with existing HAProxy expertise, it provides a familiar mental model for load balancing configuration.

Trade-offs: Smaller community than Traefik or NGINX. Less ecosystem tooling and fewer tutorials. Best suited for teams that specifically want HAProxy’s capabilities (fine-grained connection management, advanced health checking, TCP/HTTP mode flexibility) rather than as a default choice.

Kong Ingress Controller

Kong bridges the gap between an Ingress controller and a full API gateway. It supports Ingress and Gateway API resources alongside its own Kong-native plugin system for authentication, rate limiting, transformation, and observability. For teams that need API gateway capabilities rather than pure L7 routing, Kong provides a unified platform.

Trade-offs: Kong adds operational complexity. Running Kong requires either a PostgreSQL database (DB-mode) or careful management of declarative configuration (DB-less mode). The plugin ecosystem is powerful but introduces additional configuration surface. For teams that just need ingress routing, Kong may be more than necessary. For teams building API platforms, it is worth the overhead.

Istio Gateway

Istio’s ingress gateway (now aligned with Gateway API via its Kubernetes Gateway API integration) provides entry-point traffic management as part of a full service mesh. If your organization is planning or running Istio for east-west traffic, using Istio’s gateway for north-south traffic creates a unified data plane (Envoy) and consistent observability across all service communication.

Trade-offs: Istio is a serious operational commitment. The control plane overhead, the learning curve, and the impact on pod scheduling and sidecar management are significant. Choosing Istio purely for ingress replacement is like buying a race car because you needed a vehicle with good brakes. Consider this path only if service mesh capabilities are on your roadmap.

NGINX Gateway Fabric (F5’s Gateway API implementation)

F5/NGINX is building NGINX Gateway Fabric as their strategic forward path — an NGINX-based implementation of the Kubernetes Gateway API. For teams heavily invested in NGINX and wanting to stay in that ecosystem while moving to Gateway API, this provides a migration path within familiar territory. It is still maturing but represents where F5 is putting its development resources.

Comparison Matrix

Controller	Ingress API	Gateway API	Maturity	Best For	Complexity
ingress-nginx	Yes	Partial	High	Existing deployments, familiar config	Low
Traefik	Yes	Yes	High	General purpose, rapid migration	Low-Medium
Envoy Gateway	No	Yes (native)	Medium	Greenfield, future-aligned	Medium
Cilium Gateway	Yes	Yes	Medium	Cilium CNI clusters	Low (if Cilium)
HAProxy Ingress	Yes	Yes	High	High-throughput, HAProxy expertise	Medium
Kong	Yes	Yes	High	API gateway requirements	High
Istio Gateway	Via Gateway API	Yes	High	Service mesh adopters	Very High
NGINX Gateway Fabric	No	Yes (native)	Low-Medium	NGINX shops moving to Gateway API	Medium

Gateway API: The Strategic Direction You Cannot Ignore

The Kubernetes Gateway API is not simply “Ingress v2.” It is a fundamentally richer traffic management model designed to address the limitations that drove teams to annotation-based workarounds for the past several years. Understanding it is essential regardless of which controller you choose, because the ecosystem is clearly converging on it.

The core resource hierarchy consists of GatewayClass (defines a type of gateway, created by infrastructure providers), Gateway (a specific instance of a listener configuration, typically managed by platform teams), and HTTPRoute, TCPRoute, GRPCRoute, and other route resources (managed by application teams). This separation of concerns maps cleanly onto organizational roles — infrastructure teams control the gateway, application teams control their routing rules.

# Example Gateway API resources replacing an ingress-nginx Ingress
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: production-gateway
  namespace: infra
spec:
  gatewayClassName: nginx  # or envoy, traefik, cilium, etc.
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
      - name: wildcard-tls
    allowedRoutes:
      namespaces:
        from: All
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-app
  namespace: my-app-namespace
spec:
  parentRefs:
  - name: production-gateway
    namespace: infra
  hostnames:
  - "app.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: my-api-service
      port: 8080
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: my-frontend-service
      port: 3000

Gateway API reached v1.0 (GA for HTTPRoute and Gateway) in October 2023, and v1.1 followed in 2024 with GRPCRoute graduation and expanded features. The project has broad support across controllers (Traefik, Envoy Gateway, Cilium, NGINX Gateway Fabric, Kong, Istio, and others all implement it). The Ingress API is not being removed from Kubernetes, but new feature development is effectively frozen — Gateway API is where capabilities like traffic weighting, header manipulation, request mirroring, and backend protocol configuration are being built.

Decision Framework: Stay, Migrate, or Evaluate?

There is no universal right answer. The following framework helps teams make a context-appropriate decision rather than following hype or panic.

Stay on ingress-nginx if:

You have patched to 1.11.5+ or 1.12.1+ and have disabled or hardened the admission webhook
Your cluster is stable, heavily annotation-dependent, and migration cost outweighs risk
You have internal NGINX expertise and can take ownership of monitoring the project’s maintenance health
Your organization has a short-term horizon (decommissioning or major platform change within 12-18 months)

Migrate now if:

You are running the F5/NGINX IC with a support contract that is being deprecated
Your cluster has moderate annotation complexity and you have engineering cycles available
You are planning a major Kubernetes version upgrade or cluster rebuild — do it at the same time
Your security team has flagged the CVE history as unacceptable for your risk profile
You are building a new cluster or platform team and want to standardize on Gateway API from the start

Evaluate before committing if:

Your workloads have complex traffic requirements (WebSockets, gRPC, canary deployments, header-based routing) that differ significantly across controllers
You are considering Gateway API but the specific

Building a Kubernetes Migration Framework: Lessons from Ingress-NGINX

Sat, 28 Feb 2026 05:30:04 +0000

The recent announcement regarding the deprecation of the Ingress-NGINX controller sent a ripple through the Kubernetes community. For many organizations, it’s the first major deprecation of a foundational, widely-adopted ecosystem component. While the immediate reaction is often tactical—”What do we replace it with?”—the more valuable long-term question is strategic: “How do we systematically manage this and future migrations?”

This event isn’t an anomaly; it’s a precedent. As Kubernetes matures, core add-ons, APIs, and patterns will evolve or sunset. Platform engineering teams need a repeatable, low-risk framework for navigating these changes. Drawing from the Ingress-NGINX transition and established deployment management principles, we can abstract a robust Kubernetes Migration Framework applicable to any major component, from service meshes to CSI drivers.

Why Ad-Hoc Migrations Fail in Production

Attempting a “big bang” replacement or a series of manual, one-off changes is a recipe for extended downtime, configuration drift, and undetected regression. Production Kubernetes environments are complex systems with deep dependencies:

Interdependent Workloads: Multiple applications often share the same ingress controller, relying on specific annotations, custom snippets, or behavioral quirks.
Automation and GitOps Dependencies: Helm charts, Kustomize overlays, and ArgoCD/Flux manifests are tightly coupled to the existing component’s API and schema.
Observability and Security Integration: Monitoring dashboards, logging parsers, and security policies are tuned for the current implementation.
Knowledge Silos: Tribal knowledge about workarounds and specific configurations isn’t documented.

A structured framework mitigates these risks by enforcing discipline, creating clear validation gates, and ensuring the capability to roll back at any point.

The Four-Phase Kubernetes Migration Framework

This framework decomposes the migration into four distinct phases: Assessment, Parallel Run, Cutover, and Decommission. Each phase has defined inputs, activities, and exit criteria.

Phase 1: Deep Assessment & Dependency Mapping

Before writing a single line of new configuration, understand the full scope. The goal is to move from “we use Ingress-NGINX” to a precise inventory of how it’s used.

Inventory All Ingress Resources: Use kubectl get ingress --all-namespaces as a starting point, but go deeper.
Analyze Annotation Usage: Script an analysis to catalog every annotation in use (e.g., nginx.ingress.kubernetes.io/rewrite-target, nginx.ingress.kubernetes.io/configuration-snippet). This reveals functional dependencies.
Map to Backend Services: For each Ingress, identify the backend Services and Namespaces. This highlights critical applications and potential blast radius.
Review Customizations: Document any custom ConfigMaps for main NGINX configuration, custom template patches, or modifications to the controller deployment itself.
Evaluate Alternatives: Based on the inventory, evaluate candidate replacements (e.g., Gateway API with a compatible implementation, another Ingress controller like Emissary-ingress or Traefik). The Google Cloud migration framework provides a useful decision tree for ingress-specific migrations.

The output of this phase is a migration manifesto: a concrete list of what needs to be converted, grouped by complexity and criticality.

Phase 2: Phased Rollout & Parallel Run

This is the core of a low-risk migration. Instead of replacing, you run the new and old systems in parallel, shifting traffic gradually. For ingress, this often means installing the new controller alongside the old one.

Dual Installation: Deploy the new ingress controller in the same cluster, configured with a distinct ingress class (e.g., ingressClassName: gateway vs. nginx).

Create Canary Ingress Resources: For a low-risk application, create a parallel Ingress or Gateway resource pointing to the new controller. Use techniques like managed deployments with canary patterns to control exposure.

# Example: A new Gateway API HTTPRoute for a canary service
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-canary
spec:
  parentRefs:
  - name: company-gateway
  rules:
  - backendRefs:
    - name: app-service
      port: 8080
      weight: 10 # Start with 10% of traffic

Validate Equivalency: Use traffic mirroring (if supported) or direct synthetic testing against both ingress paths. Compare logs, response headers, latency, and error rates.
Iterate and Expand: Gradually increase traffic weight or add more applications to the new stack, group by group, based on the assessment from Phase 1.

This phase relies heavily on your observability stack. Dashboards comparing error rates, latency (p50, p99), and throughput between the old and new paths are essential.

Phase 3: Validation & Automated Cutover

The cutover is not a manual event. It’s the final step in a validation process.

Define Validation Tests: Create a suite of tests that must pass before full cutover. This includes:
- Smoke tests for all critical user journeys.
- Load tests to verify performance under expected traffic patterns.
- Security scan validation (e.g., no unintended ports open).
- Compliance checks (e.g., specific headers are present).
Automate the Switch: For each application, the cutover is ultimately a change in its Ingress or Gateway resource. This should be done via your GitOps pipeline. Update the source manifests (e.g., change the ingressClassName), merge, and let automation apply it. This ensures the state is declarative and recorded.
Maintain Rollback Capacity: The old system must remain operational and routable (with reduced capacity) during this phase. The GitOps rollback is simply reverting the manifest change.

Phase 4: Observability & Decommission

Once all traffic is successfully migrated and validated over a sustained period (e.g., 72 hours), you can decommission the old component.

Monitor Aggressively: Keep a close watch on all key metrics for at least one full business cycle (a week).
Remove Old Resources: Delete the old controller’s Deployment, Service, ConfigMaps, and CRDs (if no longer needed).
Clean Up Auxiliary Artifacts: Remove old RBAC bindings, service accounts, and any custom monitoring alerts or dashboards specific to the old component.
Document Lessons Learned: Update runbooks and architecture diagrams. Note any surprises, gaps in the process, or validation tests that were particularly valuable.

Key Principles for a Resilient Framework

Beyond the phases, these principles should guide your framework’s design:

Always Maintain Rollback Capability: Every step should be reversible with minimal disruption. This is a core tenet of managing Kubernetes deployments.
Leverage GitOps for State Management: All desired state changes (Ingress resources, controller deployments) must flow through version-controlled manifests. This provides an audit trail, consistency, and the simplest rollback mechanism (git revert).
Validate with Production Traffic Patterns: Synthetic tests are insufficient. Use canary weights and traffic mirroring to validate with real user traffic in a controlled manner.
Communicate Transparently: Platform teams should maintain a clear migration status page for internal stakeholders, showing which applications have been migrated, which are in progress, and the overall timeline.

Conclusion: Building a Migration-Capable Platform

The deprecation of Ingress-NGINX is a wake-up call. The next major change is a matter of “when,” not “if.” By investing in a structured migration framework now, platform teams transform a potential crisis into a manageable, repeatable operational procedure.

This framework—Assess, Run in Parallel, Validate, and Decommission—abstracts the specific lessons from the ingress migration into a generic pattern. It can be applied to migrating from PodSecurityPolicies to Pod Security Standards, from a deprecated CSI driver, or from one service mesh to another. The tools (GitOps, canary deployments, observability) are already in your stack. The value is in stitching them together into a disciplined process that ensures platform evolution doesn’t compromise platform stability.

Start by documenting this framework as a runbook template. Then, apply it to your next significant component update, even a minor one, to refine the process. When the next major deprecation announcement lands in your inbox, you’ll be ready.

Resolving Kubernetes Ingress Issues: Limitations and Gateway Insights

Fri, 13 Feb 2026 03:54:50 +0000

Introduction

Ingresses have been, since the early versions of Kubernetes, the most common way to expose applications to the outside. Although their initial design was simple and elegant, the success of Kubernetes and the growing complexity of use cases have turned Ingress into a problematic piece: limited, inconsistent between vendors, and difficult to govern in enterprise environments.

In this article, we analyze why Ingresses have become a constant source of friction, how different Ingress Controllers have influenced this situation, and why more and more organizations are considering alternatives like Gateway API.

What Ingresses are and why they were designed this way

The Ingress ecosystem revolves around two main resources:

IngressClass

Defines which controller will manage the associated Ingresses. Its scope is cluster-wide, so it is usually managed by the platform team.

Ingress

It is the resource that developers use to expose a service. It allows defining routes, domains, TLS certificates, and little more.

Its specification is minimal by design, which allowed for rapid adoption, but also laid the foundation for current problems.

The problem: a standard too simple for complex needs

As Kubernetes became an enterprise standard, users wanted to replicate advanced configurations of traditional proxies: rewrites, timeouts, custom headers, CORS, etc.
But Ingress did not provide native support for all this.

Vendors reacted… and chaos was born.

Annotations vs CRDs: two incompatible paths

Different Ingress Controllers have taken very different paths to add advanced capabilities:

Annotations (NGINX, HAProxy…)

Advantages:

Flexible and easy to use
Directly in the Ingress resource

Disadvantages:

Hundreds of proprietary annotations
Fragmented documentation
Non-portable configurations between vendors

Custom CRDs (Traefik, Kong…)

Advantages:

More structured and powerful
Better validation and control

Disadvantages:

Adds new non-standard objects
Requires installation and management
Less interoperability

Result?
Infrastructures deeply coupled to a vendor, complicating migrations, audits, and automation.

The complexity for development teams

The design of Ingress implies two very different responsibilities:

Platform: defines IngressClass
Application: defines Ingress

But the reality is that the developer ends up making decisions that should be the responsibility of the platform area:

Certificates
Security policies
Rewrite rules
CORS
Timeouts
Corporate naming practices

This causes:

Inconsistent configurations
Bottlenecks in reviews
Constant dependency between teams
Lack of effective standardization

In large companies, where security and governance are critical, this is especially problematic.

NGINX Ingress: the decommissioning that reignited the debate

The recent decommissioning of the NGINX Ingress Controller has highlighted the fragility of the ecosystem:

Thousands of clusters depend on it
Multiple projects use its annotations
Migrating involves rewriting entire configurations

This has reignited the conversation about the need for a real standard… and there appears Gateway API.

Gateway API: a promising alternative (but not perfect)

Gateway API was born to solve many of the limitations of Ingress:

Clear separation of responsibilities (infrastructure vs application)
Standardized extensibility
More types of routes (HTTPRoute, TCPRoute…)
Greater expressiveness without relying on proprietary annotations

But it also brings challenges:

Requires gradual adoption
Not all vendors implement the same
Migration is not trivial

Even so, it is shaping up to be the future of traffic management in Kubernetes.

Conclusion

Ingresses have been fundamental to the success of Kubernetes, but their own simplicity has led them to become a bottleneck. The lack of interoperability, differences between vendors, and complex governance in enterprise environments make it clear that it is time to adopt more mature models.

Gateway API is not perfect, but it moves in the right direction.
Organizations that want future stability should start planning their transition.

Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.