<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0">
<channel>
<title>Coder Legion - Recent posts tagged container</title>
<link>https://coderlegion.com/tag/container</link>
<description>Powered by Question2Answer</description>
<item>
<title>Kaniko, BuildKit, and Image Volumes: The Evolution of Container Images Inside Kubernetes</title>
<link>https://coderlegion.com/23252/kaniko-buildkit-and-image-volumes-the-evolution-of-container-images-inside-kubernetes</link>
<description>
&lt;a href=&quot;https://alexandre-vazquez.com/kaniko-image-volumes-kubernetes-build-history/&quot; title=&quot;Kaniko, BuildKit, and Image Volumes: The Evolution of Container Images Inside Kubernetes&quot; rel=&quot;nofollow&quot;&gt;&lt;img width=&quot;1536&quot; height=&quot;1024&quot; src=&quot;https://mlk5tkvgfeoa.i.optimole.com/cb:s8WW.68/w:1536/h:1024/q:mauto/f:best/https://alexandre-vazquez.com/wp-content/uploads/2026/07/kaniko.avif&quot; class=&quot;webfeedsFeaturedVisual wp-post-image&quot; alt=&quot;Kaniko, BuildKit, and Image Volumes: The Evolution of Container Images Inside Kubernetes&quot; style=&quot;display: block; margin: auto; margin-bottom: 5px;max-width: 100%;&quot; title=&quot;Kaniko, BuildKit, and Image Volumes: The Evolution of Container Images Inside Kubernetes 8&quot;&gt;&lt;/a&gt;
&lt;p&gt;Running container images inside Kubernetes is table stakes. &lt;em&gt;Building&lt;/em&gt; them there &amp;mdash; or mounting their contents as volumes &amp;mdash; has been a moving target for years. What started as a privileged hack has evolved into a set of mature, secure, and increasingly native primitives.&lt;/p&gt;



&lt;p&gt;This article covers the full arc: why the original approach was broken, what Kaniko solved, why its maintenance status now matters, where BuildKit and Buildah fit, and what Image Volumes (stable in Kubernetes v1.36) change for workflows that don&amp;rsquo;t need to build anything at all.&lt;/p&gt;



&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot;&gt;



&lt;h2 class=&quot;wp-block-heading&quot;&gt;The original problem: Docker-in-Docker&lt;/h2&gt;



&lt;p&gt;The first generation of CI/CD on Kubernetes ran Docker inside Docker. You mounted the host Docker socket (&lt;code&gt;/var/run/docker.sock&lt;/code&gt;) into a build container, and that container had full access to the host&amp;rsquo;s Docker daemon.&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;# The approach nobody should be using in 2026
volumes:
- name: docker-sock
  hostPath:
    path: /var/run/docker.sock&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;This worked. It was also a complete security disaster.&lt;/p&gt;



&lt;p&gt;Mounting the Docker socket gives the container root-equivalent access to the host. Any workload that can reach that socket can escape the container, inspect other containers, and compromise the node. The only thing standing between your CI pipeline and a full cluster compromise was the good intentions of whoever wrote the build script.&lt;/p&gt;



&lt;p&gt;It got worse when the Kubernetes project deprecated Dockershim in v1.20 and removed it in v1.24. Clusters that moved to containerd or CRI-O no longer had a Docker daemon on the host at all. The socket either didn&amp;rsquo;t exist or belonged to a completely different runtime. Docker-in-Docker in its classic form became functionally impossible on modern clusters.&lt;/p&gt;



&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot;&gt;



&lt;h2 class=&quot;wp-block-heading&quot;&gt;Kaniko: daemonless builds inside containers&lt;/h2&gt;



&lt;p&gt;Google released Kaniko in 2018 to solve exactly this problem. Kaniko builds container images entirely in userspace &amp;mdash; no daemon, no privileged access, no host socket required.&lt;/p&gt;



&lt;p&gt;The key insight: Docker builds work by executing each &lt;code&gt;RUN&lt;/code&gt; instruction in a temporary container, snapshotting the filesystem, and saving the result as a layer. Kaniko replicates this logic without a daemon. It runs as a regular container, reads the Dockerfile, executes each step against the local filesystem, and &lt;a rel=&quot;nofollow&quot; href=&quot;https://alexandre-vazquez.com/skopeo-crane-regctl-container-image-tools/&quot;&gt;pushes the resulting image directly to a registry&lt;/a&gt;.&lt;/p&gt;



&lt;p&gt;That design aged well. The project governance did not. The &lt;code&gt;GoogleContainerTools/kaniko&lt;/code&gt; repository was archived by its owner on June 3, 2025 and is now read-only. In practical terms, the original Google-hosted Kaniko project should be treated as unmaintained: no active upstream issue triage, no normal pull request flow, and no clear path for security fixes through that repository.&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;apiVersion: v1
kind: Pod
metadata:
  name: kaniko-build
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args:
    - &quot;--dockerfile=Dockerfile&quot;
    - &quot;--context=git://github.com/your-org/your-repo&quot;
    - &quot;--destination=your-registry/your-image:tag&quot;
    volumeMounts:
    - name: registry-creds
      mountPath: /kaniko/.docker
  volumes:
  - name: registry-creds
    secret:
      secretName: registry-credentials
      items:
      - key: .dockerconfigjson
        path: config.json
  restartPolicy: Never&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;No host socket. No privileged flag. The container needs write access to its own filesystem (so &lt;code&gt;readOnlyRootFilesystem: true&lt;/code&gt; won&amp;rsquo;t work), but that&amp;rsquo;s a far narrower requirement than socket mounting.&lt;/p&gt;



&lt;h3 class=&quot;wp-block-heading&quot;&gt;When Kaniko is still a reasonable answer&lt;/h3&gt;



&lt;p&gt;Kaniko can still be a reasonable tactical choice when you need to &lt;strong&gt;build&lt;/strong&gt; a container image inside Kubernetes and already have a working, isolated pipeline around it. Existing Kaniko jobs did not stop working when the repository was archived.&lt;/p&gt;



&lt;p&gt;For new platform work in 2026, though, do not pick Kaniko by default. The maintenance signal changes the risk model. Use it only if the simplicity is worth owning the upgrade and vulnerability-management story yourself, or if you deliberately standardize on a maintained fork or vendor-supported distribution.&lt;/p&gt;



&lt;p&gt;Kaniko integrates well with: &amp;ndash; &lt;strong&gt;Tekton&lt;/strong&gt; &amp;mdash; the standard pattern is a Tekton Task running the Kaniko executor &amp;ndash; &lt;strong&gt;Argo Workflows&lt;/strong&gt; &amp;mdash; same pattern, different orchestrator &amp;ndash; &lt;strong&gt;GitLab CI on Kubernetes&lt;/strong&gt; &amp;mdash; many older examples and pipelines use Kaniko with the Kubernetes executor &amp;ndash; &lt;strong&gt;Any pod-based CI system&lt;/strong&gt; &amp;mdash; it&amp;rsquo;s just a container, so it runs anywhere pods run&lt;/p&gt;



&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot;&gt;



&lt;h2 class=&quot;wp-block-heading&quot;&gt;The alternatives: BuildKit and Buildah&lt;/h2&gt;



&lt;p&gt;Kaniko is no longer the only serious daemonless option. Two other tools are worth evaluating first for new work:&lt;/p&gt;



&lt;h3 class=&quot;wp-block-heading&quot;&gt;BuildKit (rootless mode)&lt;/h3&gt;



&lt;p&gt;BuildKit is the build backend behind &lt;code&gt;docker buildx&lt;/code&gt; and the default build system in modern Docker. In rootless mode it runs without privileges and can build images inside a Kubernetes pod.&lt;/p&gt;



&lt;p&gt;BuildKit has better caching than Kaniko &amp;mdash; particularly layer caching via cache mounts &amp;mdash; and supports more advanced Dockerfile features like heredocs and multi-platform builds. The tradeoff is more complex setup: you need to run &lt;code&gt;buildkitd&lt;/code&gt; as a sidecar or as a DaemonSet.&lt;/p&gt;



&lt;p&gt;For teams already using &lt;code&gt;docker buildx&lt;/code&gt; locally, BuildKit is the most natural migration path. Docker&amp;rsquo;s Kubernetes driver can run BuildKit builders directly in a cluster, including rootless mode without privileged pods on supported Kubernetes versions. The operational cost is real &amp;mdash; builder lifecycle, cache persistence, node placement, and rootless kernel requirements &amp;mdash; but the project is active and the feature set is where most modern Dockerfile workflows are moving.&lt;/p&gt;



&lt;h3 class=&quot;wp-block-heading&quot;&gt;Buildah&lt;/h3&gt;



&lt;p&gt;Buildah is the containers project&amp;rsquo;s daemonless build tool, designed to integrate with Podman and OCI-native workflows. It can build from Dockerfiles or Containerfiles, and it is the natural choice on OpenShift or in environments where the platform team already standardizes on Red Hat, Podman, and the &lt;code&gt;containers/*&lt;/code&gt; stack.&lt;/p&gt;



&lt;p&gt;The caveat is that rootless Buildah inside a restricted Kubernetes pod still depends on user namespace behavior and helper binaries such as &lt;code&gt;newuidmap&lt;/code&gt; and &lt;code&gt;newgidmap&lt;/code&gt;. That is manageable on platforms built for it, especially OpenShift, but it is not automatically a drop-in replacement for every generic Kubernetes CI runner.&lt;/p&gt;



&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot;&gt;



&lt;h2 class=&quot;wp-block-heading&quot;&gt;Image Volumes: a different problem entirely&lt;/h2&gt;



&lt;p&gt;Here is where the narrative splits. Everything above is about &lt;strong&gt;building&lt;/strong&gt; images. Image Volumes solve a completely different problem: &lt;strong&gt;consuming the contents of an OCI image as a volume&lt;/strong&gt;, without building anything.&lt;/p&gt;



&lt;p&gt;The feature was introduced as alpha in Kubernetes v1.31, moved to beta in v1.33 with &lt;code&gt;subPath&lt;/code&gt; and &lt;code&gt;subPathExpr&lt;/code&gt; support, became beta enabled by default in v1.35, and graduated to stable (GA) in Kubernetes v1.36.&lt;/p&gt;



&lt;h3 class=&quot;wp-block-heading&quot;&gt;What it does&lt;/h3&gt;



&lt;p&gt;Image Volumes let you reference an OCI image in a pod&amp;rsquo;s &lt;code&gt;volumes&lt;/code&gt; section and mount its filesystem contents directly into a container:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;apiVersion: v1
kind: Pod
metadata:
  name: image-volume-example
spec:
  containers:
  - name: app
    image: debian
    command: [&quot;sleep&quot;, &quot;infinity&quot;]
    volumeMounts:
    - name: config-data
      mountPath: /app/config
  volumes:
  - name: config-data
    image:
      reference: your-registry/your-config-image:v1.2.0
      pullPolicy: IfNotPresent&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;The container at &lt;code&gt;/app/config&lt;/code&gt; sees the contents of &lt;code&gt;your-config-image:v1.2.0&lt;/code&gt;. The volume is read-only. No init container required.&lt;/p&gt;



&lt;p&gt;From Kubernetes v1.33+, you can also mount a subdirectory with &lt;code&gt;subPath&lt;/code&gt;:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;volumeMounts:
- name: config-data
  mountPath: /app/config
  subPath: environments/production&lt;/code&gt;&lt;/pre&gt;



&lt;h3 class=&quot;wp-block-heading&quot;&gt;The use case: OCI images as artifact bundles&lt;/h3&gt;



&lt;p&gt;This feature is motivated by a pattern that has been growing quietly: using OCI images not as runnable containers but as versioned, signed, distributable artifact bundles.&lt;/p&gt;



&lt;p&gt;The idea: instead of storing configuration, schemas, WASM modules, ML models, or static binaries in a ConfigMap or a separate volume, you package them as an OCI image. You get:&lt;/p&gt;



&lt;ul class=&quot;wp-block-list&quot;&gt;

&lt;li&gt;&lt;strong&gt;Version control&lt;/strong&gt; &amp;mdash; image tags and digests, same tooling you already use&lt;/li&gt;


&lt;li&gt;&lt;strong&gt;Distribution&lt;/strong&gt; &amp;mdash; your existing registry, your existing pull secrets, your existing access controls&lt;/li&gt;


&lt;li&gt;&lt;strong&gt;Signing and attestation&lt;/strong&gt; &amp;mdash; cosign, Sigstore, the full supply chain tooling works on these artifacts&lt;/li&gt;


&lt;li&gt;&lt;strong&gt;Immutability&lt;/strong&gt; &amp;mdash; a digest-pinned image reference is cryptographically immutable&lt;/li&gt;

&lt;/ul&gt;



&lt;p&gt;Image Volumes are the Kubernetes primitive that makes this pattern first-class. Without Image Volumes, the workaround was an init container that pulled the image and copied the contents to an emptyDir. It worked, but it was boilerplate.&lt;/p&gt;



&lt;h3 class=&quot;wp-block-heading&quot;&gt;What it does not do&lt;/h3&gt;



&lt;p&gt;Image Volumes are not a replacement for Kaniko or any build tool. They consume images; they don&amp;rsquo;t produce them. If your workflow involves building a new image from source, you still need Kaniko, BuildKit, or equivalent.&lt;/p&gt;



&lt;p&gt;They also require container runtime support. CRI-O supported the initial alpha implementation from its v1.31 line and tracked beta support for v1.33; containerd support landed later through the containerd 2.x line. On anything before Kubernetes v1.36, verify both the Kubernetes feature gate and the runtime version before treating this as production plumbing.&lt;/p&gt;



&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot;&gt;



&lt;h2 class=&quot;wp-block-heading&quot;&gt;Decision framework by Kubernetes version&lt;/h2&gt;



&lt;figure class=&quot;wp-block-table&quot;&gt;&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;K8s version&lt;/th&gt;&lt;th&gt;Build images (CI)&lt;/th&gt;&lt;th&gt;Mount image contents as volume&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;&amp;lt; 1.24&lt;/td&gt;&lt;td&gt;Avoid Docker socket builds; use BuildKit, Buildah, or legacy Kaniko with explicit risk acceptance&lt;/td&gt;&lt;td&gt;Init container + &lt;code&gt;emptyDir&lt;/code&gt; workaround&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;1.24 &amp;ndash; 1.30&lt;/td&gt;&lt;td&gt;BuildKit or Buildah preferred; legacy Kaniko only if already standardized&lt;/td&gt;&lt;td&gt;Init container + &lt;code&gt;emptyDir&lt;/code&gt; workaround&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;1.31 &amp;ndash; 1.32&lt;/td&gt;&lt;td&gt;BuildKit or Buildah preferred; legacy Kaniko only with maintenance plan&lt;/td&gt;&lt;td&gt;Image Volumes alpha &amp;mdash; &lt;code&gt;ImageVolume&lt;/code&gt; feature gate required, runtime support required&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;1.33 &amp;ndash; 1.34&lt;/td&gt;&lt;td&gt;BuildKit or Buildah preferred; legacy Kaniko only with maintenance plan&lt;/td&gt;&lt;td&gt;Image Volumes beta with &lt;code&gt;subPath&lt;/code&gt;/&lt;code&gt;subPathExpr&lt;/code&gt;, but disabled by default; enable &lt;code&gt;ImageVolume&lt;/code&gt; and verify runtime support&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;1.35&lt;/td&gt;&lt;td&gt;BuildKit or Buildah preferred; legacy Kaniko only with maintenance plan&lt;/td&gt;&lt;td&gt;Image Volumes beta, enabled by default; still verify runtime support before production rollout&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;1.36+&lt;/td&gt;&lt;td&gt;BuildKit or Buildah preferred; legacy Kaniko only for existing pipelines or maintained forks&lt;/td&gt;&lt;td&gt;Image Volumes stable (GA), enabled by default&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/figure&gt;



&lt;h3 class=&quot;wp-block-heading&quot;&gt;When to use what&lt;/h3&gt;



&lt;p&gt;&lt;strong&gt;Use Kaniko if:&lt;/strong&gt; &amp;ndash; You already have working Kaniko pipelines and the operational cost of migration is higher than the current risk &amp;ndash; You have a maintained fork, vendor support, or an internal patching process &amp;ndash; You want the simplest daemonless build setup and accept that upstream Google Kaniko is archived&lt;/p&gt;



&lt;p&gt;&lt;strong&gt;Use BuildKit (rootless) if:&lt;/strong&gt; &amp;ndash; You need advanced cache mounts or multi-platform builds &amp;ndash; Your team already uses &lt;code&gt;docker buildx&lt;/code&gt; locally &amp;ndash; You&amp;rsquo;re willing to run and operate BuildKit builders in the cluster&lt;/p&gt;



&lt;p&gt;&lt;strong&gt;Use Buildah if:&lt;/strong&gt; &amp;ndash; You are on OpenShift or a Podman/Red Hat-oriented platform &amp;ndash; You want an OCI-native build tool that does not require a daemon &amp;ndash; Your cluster policy supports the user namespace requirements of rootless builds&lt;/p&gt;



&lt;p&gt;&lt;strong&gt;Use Image Volumes if:&lt;/strong&gt; &amp;ndash; You want to inject versioned, signed artifacts into pods without building anything &amp;ndash; You&amp;rsquo;re replacing init container + emptyDir patterns &amp;ndash; You&amp;rsquo;re adopting OCI images as a general artifact format (configs, schemas, binaries)&lt;/p&gt;



&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot;&gt;



&lt;h2 class=&quot;wp-block-heading&quot;&gt;Conclusion&lt;/h2&gt;



&lt;p&gt;The container image story inside Kubernetes has matured significantly. Docker-in-Docker is dead &amp;mdash; correctly so. Kaniko solved an important build problem in 2018, but the original Google project is archived in 2026, so it should no longer be the default recommendation for new platforms. BuildKit and Buildah are the healthier starting points for active build pipelines, with Kaniko reserved for existing estates or explicitly supported forks.&lt;/p&gt;



&lt;p&gt;Image Volumes are genuinely new ground. They&amp;rsquo;re not competing with Kaniko &amp;mdash; they&amp;rsquo;re addressing a different layer of the same ecosystem: the distribution and consumption of OCI artifacts beyond just &amp;ldquo;images you run.&amp;rdquo; With GA in v1.36 and the supply chain tooling around OCI reaching maturity, the pattern of &amp;ldquo;package it as an image, distribute it like an image, mount it where you need it&amp;rdquo; is becoming the right answer for a class of problems that previously lived in ConfigMaps, PVCs, or init container hacks.&lt;/p&gt;



&lt;p&gt;The right combination depends on what you&amp;rsquo;re doing and what version you&amp;rsquo;re running. But for the first time, Kubernetes has native answers for both sides of the equation.&lt;/p&gt;



&lt;h2 class=&quot;wp-block-heading&quot;&gt;Choose today&lt;/h2&gt;



&lt;p&gt;If you operate Kubernetes v1.36 or newer, use Image Volumes for read-only artifact injection and choose BuildKit or Buildah for builds. If you are on v1.35, Image Volumes are beta and enabled by default, but you should still verify runtime support and keep the init-container pattern as the rollback path. If you are on v1.33 or v1.34, beta does not mean default-on: enable the &lt;code&gt;ImageVolume&lt;/code&gt; feature gate deliberately and validate the runtime. If you are on v1.31 or v1.32, treat Image Volumes as alpha and non-default. If you are below v1.31, Image Volumes are not part of the platform: use &lt;code&gt;emptyDir&lt;/code&gt; plus an init container for artifact mounting, and modernize the build path separately.&lt;/p&gt;



&lt;p&gt;For CI builds, start with BuildKit when you want Dockerfile compatibility, cache performance, multi-platform output, and a path aligned with &lt;code&gt;docker buildx&lt;/code&gt;. Start with Buildah when your cluster is OpenShift or Podman-oriented. Keep Kaniko only where it already works and where someone owns the maintenance risk.&lt;/p&gt;



&lt;h2 class=&quot;wp-block-heading&quot;&gt;Sources&lt;/h2&gt;



&lt;ul class=&quot;wp-block-list&quot;&gt;

&lt;li&gt;&lt;a href=&quot;https://github.com/GoogleContainerTools/kaniko&quot; rel=&quot;nofollow&quot;&gt;https://github.com/GoogleContainerTools/kaniko&lt;/a&gt;&lt;/li&gt;


&lt;li&gt;&lt;a href=&quot;https://github.com/kubernetes/enhancements/issues/4639&quot; rel=&quot;nofollow&quot;&gt;https://github.com/kubernetes/enhancements/issues/4639&lt;/a&gt;&lt;/li&gt;


&lt;li&gt;&lt;a href=&quot;https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/&quot; rel=&quot;nofollow&quot;&gt;https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/&lt;/a&gt;&lt;/li&gt;


&lt;li&gt;&lt;a href=&quot;https://kubernetes.io/blog/2024/08/16/kubernetes-1-31-image-volume-source/&quot; rel=&quot;nofollow&quot;&gt;https://kubernetes.io/blog/2024/08/16/kubernetes-1-31-image-volume-source/&lt;/a&gt;&lt;/li&gt;


&lt;li&gt;&lt;a href=&quot;https://kubernetes.io/blog/2025/04/29/kubernetes-v1-33-image-volume-beta/&quot; rel=&quot;nofollow&quot;&gt;https://kubernetes.io/blog/2025/04/29/kubernetes-v1-33-image-volume-beta/&lt;/a&gt;&lt;/li&gt;


&lt;li&gt;&lt;a href=&quot;https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/&quot; rel=&quot;nofollow&quot;&gt;https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/&lt;/a&gt;&lt;/li&gt;


&lt;li&gt;&lt;a href=&quot;https://docs.docker.com/build/builders/drivers/kubernetes/&quot; rel=&quot;nofollow&quot;&gt;https://docs.docker.com/build/builders/drivers/kubernetes/&lt;/a&gt;&lt;/li&gt;


&lt;li&gt;&lt;a href=&quot;https://github.com/moby/buildkit/blob/master/docs/rootless.md&quot; rel=&quot;nofollow&quot;&gt;https://github.com/moby/buildkit/blob/master/docs/rootless.md&lt;/a&gt;&lt;/li&gt;


&lt;li&gt;&lt;a href=&quot;https://github.com/containers/buildah&quot; rel=&quot;nofollow&quot;&gt;https://github.com/containers/buildah&lt;/a&gt;&lt;/li&gt;


&lt;li&gt;&lt;a href=&quot;https://github.com/containers/buildah/blob/main/docs/tutorials/05-openshift-rootless-build.md&quot; rel=&quot;nofollow&quot;&gt;https://github.com/containers/buildah/blob/main/docs/tutorials/05-openshift-rootless-build.md&lt;/a&gt;&lt;/li&gt;

&lt;/ul&gt;
&lt;div class=&quot;yarpp yarpp-related yarpp-related-rss yarpp-template-list&quot;&gt;
&amp;lt;!-- YARPP List --&amp;gt;
&lt;h2&gt;Related articles:&lt;/h2&gt;&lt;ol&gt;
&lt;li&gt;&lt;a href=&quot;https://alexandre-vazquez.com/skopeo-crane-regctl-container-image-tools/&quot; rel=&quot;bookmark nofollow&quot; title=&quot;Skopeo, Crane, and regctl: Container Image Management Without the Docker Daemon (2026)&quot;&gt;Skopeo, Crane, and regctl: Container Image Management Without the Docker Daemon (2026)&lt;/a&gt; &lt;small&gt;Skopeo, crane, regctl, ORAS &amp;mdash; a practical comparison of container...&lt;/small&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://alexandre-vazquez.com/debugging-distroless-containers/&quot; rel=&quot;bookmark nofollow&quot; title=&quot;Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each&quot;&gt;Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each&lt;/a&gt; &lt;small&gt;The container works fine in CI. It deploys successfully to...&lt;/small&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://alexandre-vazquez.com/talos-linux-guide/&quot; rel=&quot;bookmark nofollow&quot; title=&quot;Talos Linux: The Immutable, API-Driven OS for Kubernetes (Deep Dive)&quot;&gt;Talos Linux: The Immutable, API-Driven OS for Kubernetes (Deep Dive)&lt;/a&gt; &lt;small&gt;Talos Linux strips away everything a Kubernetes node does not...&lt;/small&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</description>
<category>Other</category>
<guid isPermaLink="true">https://coderlegion.com/23252/kaniko-buildkit-and-image-volumes-the-evolution-of-container-images-inside-kubernetes</guid>
<pubDate>Sun, 26 Jul 2026 10:00:00 +0000</pubDate>
</item>
<item>
<title>Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each</title>
<link>https://coderlegion.com/15642/debugging-distroless-containers-kubectl-debug-ephemeral-containers-and-when-to-use-each</link>
<description>
&lt;a href=&quot;https://alexandre-vazquez.com/debugging-distroless-containers/&quot; title=&quot;Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each&quot; rel=&quot;nofollow&quot;&gt;&lt;img width=&quot;1536&quot; height=&quot;1024&quot; src=&quot;https://mlk5tkvgfeoa.i.optimole.com/cb:s8WW.68/w:1536/h:1024/q:mauto/f:best/https://alexandre-vazquez.com/wp-content/uploads/2026/04/debugging-distroless-containers.png&quot; class=&quot;webfeedsFeaturedVisual wp-post-image&quot; alt=&quot;Developer inspecting a distroless container with magnifying glass&quot; style=&quot;display: block; margin: auto; margin-bottom: 5px;max-width: 100%;&quot; sizes=&quot;(max-width: 1536px) 100vw, 1536px&quot; title=&quot;Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each 6&quot;&gt;&lt;/a&gt;&amp;lt;!--
SEO Title: Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each
Meta Description: Distroless containers have no shell, no tools, no debugger. Learn every technique to debug them: kubectl debug, ephemeral containers, copy-to strategy, cdebug, and node-level access &amp;mdash; with RBAC patterns for dev and production.
Focus Keywords: kubectl debug distroless,debug distroless container kubernetes,distroless container debugging,kubectl debug ephemeral container
Suggested Slug: debugging-distroless-containers
--&amp;gt;


&lt;p&gt;The container works fine in CI. It deploys successfully to staging. Then something goes wrong in production and you type the command you always type: &lt;code&gt;kubectl exec -it my-pod -- /bin/bash&lt;/code&gt;. The response is immediate: &lt;code&gt;OCI runtime exec failed: exec failed: unable to start container process: exec: &quot;/bin/bash&quot;: stat /bin/bash: no such file or directory&lt;/code&gt;.&lt;/p&gt;



&lt;p&gt;You try &lt;code&gt;/bin/sh&lt;/code&gt;. Same error. You try &lt;code&gt;ls&lt;/code&gt;. Same error. The container image is distroless &amp;mdash; it ships only your application binary and its runtime dependencies, with no shell, no package manager, no debugging tools of any kind. This is intentional and correct from a security standpoint. It is also a significant operational challenge the first time you face it in production.&lt;/p&gt;



&lt;p&gt;This article covers every practical technique for debugging distroless containers in Kubernetes: &lt;strong&gt;kubectl debug with ephemeral containers&lt;/strong&gt; (the standard approach), &lt;strong&gt;pod copy strategy&lt;/strong&gt; (for Kubernetes versions without ephemeral container support, or when you need to modify the running pod spec), &lt;strong&gt;debug image variants&lt;/strong&gt; (the pragmatic developer shortcut), &lt;strong&gt;cdebug&lt;/strong&gt; (a purpose-built tool that simplifies the process), and &lt;strong&gt;node-level debugging&lt;/strong&gt; (the last resort with the most power). For each technique I will explain what it can and cannot do, what Kubernetes version or RBAC permissions it requires, and in which scenario &amp;mdash; developer in local, platform engineer in staging, ops in production &amp;mdash; it is the appropriate choice.&lt;/p&gt;



&lt;h2 class=&quot;wp-block-heading&quot;&gt;Why Distroless Breaks the Normal Debugging Workflow&lt;/h2&gt;



&lt;p&gt;Traditional container debugging assumes you can exec into the container and use shell tools: &lt;code&gt;ps&lt;/code&gt;, &lt;code&gt;netstat&lt;/code&gt;, &lt;code&gt;strace&lt;/code&gt;, &lt;code&gt;curl&lt;/code&gt;, a text editor. Distroless images remove all of this by design. The Google distroless project, Chainguard&amp;rsquo;s Wolfi-based images, and the broader minimal image ecosystem deliberately exclude everything that is not required to run the application. The result is a dramatically smaller attack surface: no shell means no RCE via shell injection, no package manager means no easy escalation path, fewer binaries means fewer CVEs in the image scan.&lt;/p&gt;



&lt;p&gt;The tradeoff is operational: when something goes wrong, you cannot use the tools that the process itself is not allowed to run. A Java application in &lt;code&gt;gcr.io/distroless/java17-debian12&lt;/code&gt; has the JRE and nothing else. A Go binary compiled with CGO disabled and shipped in &lt;code&gt;gcr.io/distroless/static-debian12&lt;/code&gt; has literally only the binary and the necessary CA certificates and timezone data. There is no &lt;code&gt;wget&lt;/code&gt; to download a debug binary, no &lt;code&gt;apt&lt;/code&gt; to install one, no &lt;code&gt;bash&lt;/code&gt; to run a script.&lt;/p&gt;



&lt;p&gt;Kubernetes solves this at the platform level with &lt;strong&gt;ephemeral containers&lt;/strong&gt;, added as stable in Kubernetes 1.25. The principle is that a debug container &amp;mdash; which can have a full shell and any tools you want &amp;mdash; can be injected into a running pod and share its process namespace, network namespace, and filesystem mounts without modifying the original container or restarting the pod.&lt;/p&gt;



&lt;h2 class=&quot;wp-block-heading&quot;&gt;Option 1: kubectl debug with Ephemeral Containers&lt;/h2&gt;



&lt;p&gt;Ephemeral containers are the canonical solution. Since Kubernetes 1.25 (stable), &lt;code&gt;kubectl debug&lt;/code&gt; can inject a temporary container into a running pod. The container shares the target pod&amp;rsquo;s network namespace by default, and with &lt;code&gt;--target&lt;/code&gt; it can also share the process namespace of a specific container, allowing you to inspect its running processes and open file descriptors.&lt;/p&gt;



&lt;p&gt;The basic invocation is:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;kubectl debug -it my-pod \
  --image=busybox:latest \
  --target=my-container&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;The &lt;code&gt;--target&lt;/code&gt; flag is the critical piece. Without it, the ephemeral container gets its own process namespace. With it, it shares the process namespace of the specified container &amp;mdash; meaning you can run &lt;code&gt;ps aux&lt;/code&gt; and see the application&amp;rsquo;s processes, use &lt;code&gt;ls -la /proc/&amp;lt;pid&amp;gt;/fd&lt;/code&gt; to inspect open file descriptors, and read the application&amp;rsquo;s environment via &lt;code&gt;cat /proc/&amp;lt;pid&amp;gt;/environ&lt;/code&gt;.&lt;/p&gt;



&lt;p&gt;For a more capable debug environment, replace &lt;code&gt;busybox&lt;/code&gt; with a richer image:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;kubectl debug -it my-pod \
  --image=nicolaka/netshoot \
  --target=my-container&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;&lt;code&gt;nicolaka/netshoot&lt;/code&gt; includes &lt;code&gt;tcpdump&lt;/code&gt;, &lt;code&gt;curl&lt;/code&gt;, &lt;code&gt;dig&lt;/code&gt;, &lt;code&gt;nmap&lt;/code&gt;, &lt;code&gt;ss&lt;/code&gt;, &lt;code&gt;iperf3&lt;/code&gt;, and dozens of other network diagnostic tools, making it the standard choice for network debugging scenarios.&lt;/p&gt;



&lt;h3 class=&quot;wp-block-heading&quot;&gt;What You Can and Cannot Do&lt;/h3&gt;



&lt;p&gt;Ephemeral containers share the pod&amp;rsquo;s network namespace and, when &lt;code&gt;--target&lt;/code&gt; is used, the process namespace. This gives you:&lt;/p&gt;



&lt;ul class=&quot;wp-block-list&quot;&gt;
&lt;li&gt;Full visibility into the application&amp;rsquo;s network traffic from inside the pod (tcpdump, ss, netstat)&lt;/li&gt;
&lt;li&gt;Process inspection via &lt;code&gt;/proc/&amp;lt;pid&amp;gt;&lt;/code&gt; &amp;mdash; open files, memory maps, environment variables, CPU/memory usage&lt;/li&gt;
&lt;li&gt;Access to the pod&amp;rsquo;s DNS resolution context &amp;mdash; exactly the same &lt;code&gt;/etc/resolv.conf&lt;/code&gt; the application sees&lt;/li&gt;
&lt;li&gt;Ability to make outbound network calls from the same network namespace (testing service endpoints, DNS resolution)&lt;/li&gt;
&lt;/ul&gt;



&lt;p&gt;What you do &lt;em&gt;not&lt;/em&gt; get with ephemeral containers:&lt;/p&gt;



&lt;ul class=&quot;wp-block-list&quot;&gt;
&lt;li&gt;&lt;strong&gt;Access to the application container&amp;rsquo;s filesystem.&lt;/strong&gt; The ephemeral container has its own root filesystem. You cannot &lt;code&gt;cat /app/config.yaml&lt;/code&gt; from the application container&amp;rsquo;s filesystem unless you access it via &lt;code&gt;/proc/&amp;lt;pid&amp;gt;/root/&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Ability to remove the container once added.&lt;/strong&gt; Ephemeral containers are permanent until the pod is deleted. This is by design &amp;mdash; the Kubernetes API does not allow removing them after creation.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Volume mount modifications via CLI.&lt;/strong&gt; You cannot add volume mounts to an ephemeral container via &lt;code&gt;kubectl debug&lt;/code&gt; (though the API spec supports it, the CLI does not expose this).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Resource limits.&lt;/strong&gt; Ephemeral containers do not support resource requests and limits in the &lt;code&gt;kubectl debug&lt;/code&gt; CLI, though this is evolving.&lt;/li&gt;
&lt;/ul&gt;



&lt;h3 class=&quot;wp-block-heading&quot;&gt;Accessing the Application Filesystem&lt;/h3&gt;



&lt;p&gt;The most common surprise for developers new to ephemeral containers is that they cannot directly browse the application container&amp;rsquo;s filesystem. The workaround is the &lt;code&gt;/proc&lt;/code&gt; filesystem:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;# Find the application's PID
ps aux

# Browse its filesystem via /proc
ls /proc/1/root/app/
cat /proc/1/root/etc/config.yaml

# Or set the root to the application's root
chroot /proc/1/root /bin/sh  # only if /bin/sh exists in the app image&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;The &lt;code&gt;/proc/&amp;lt;pid&amp;gt;/root&lt;/code&gt; path is a symlink to the container&amp;rsquo;s root filesystem as seen from the process namespace. Because the ephemeral container shares the process namespace with &lt;code&gt;--target&lt;/code&gt;, the application&amp;rsquo;s PID is typically 1, and &lt;code&gt;/proc/1/root&lt;/code&gt; gives you full read access to its filesystem.&lt;/p&gt;



&lt;h3 class=&quot;wp-block-heading&quot;&gt;RBAC Requirements&lt;/h3&gt;



&lt;p&gt;Ephemeral containers require the &lt;code&gt;pods/ephemeralcontainers&lt;/code&gt; subresource permission. This is separate from &lt;code&gt;pods/exec&lt;/code&gt;, which controls &lt;code&gt;kubectl exec&lt;/code&gt;. A common mistake is to grant &lt;code&gt;pods/exec&lt;/code&gt; for debugging purposes without realizing that ephemeral containers require an additional grant:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ephemeral-debugger
rules:
- apiGroups: [&quot;&quot;]
  resources: [&quot;pods/ephemeralcontainers&quot;]
  verbs: [&quot;update&quot;, &quot;patch&quot;]
- apiGroups: [&quot;&quot;]
  resources: [&quot;pods/attach&quot;]
  verbs: [&quot;create&quot;, &quot;get&quot;]
- apiGroups: [&quot;&quot;]
  resources: [&quot;pods&quot;]
  verbs: [&quot;get&quot;, &quot;list&quot;]&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;In production environments, this permission should be tightly scoped: time-limited via &lt;code&gt;RoleBinding&lt;/code&gt; rather than permanent &lt;code&gt;ClusterRoleBinding&lt;/code&gt;, restricted to specific namespaces, and ideally gated behind an approval workflow. The debug container runs as root by default, which can create privilege escalation paths if the application container runs as a non-root user with shared process namespace &amp;mdash; the debug container can attach to the application&amp;rsquo;s processes with higher privileges.&lt;/p&gt;



&lt;h2 class=&quot;wp-block-heading&quot;&gt;Option 2: kubectl debug &amp;ndash;copy-to (Pod Copy Strategy)&lt;/h2&gt;



&lt;p&gt;When you need to modify the pod&amp;rsquo;s container spec &amp;mdash; replace the image, change environment variables, add a sidecar with a shared filesystem &amp;mdash; the &lt;code&gt;--copy-to&lt;/code&gt; flag creates a full copy of the pod with your modifications applied:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;kubectl debug my-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=my-app:debug \
  --share-processes&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;This creates a new pod named &lt;code&gt;my-pod-debug&lt;/code&gt; that is a copy of &lt;code&gt;my-pod&lt;/code&gt; but with the container image replaced by &lt;code&gt;my-app:debug&lt;/code&gt;. If &lt;code&gt;my-app:debug&lt;/code&gt; is your application image built with debug tooling included (or a debug variant from your registry), this lets you interact with the exact same binary in the exact same configuration as the original pod.&lt;/p&gt;



&lt;p&gt;A more common use of &lt;code&gt;--copy-to&lt;/code&gt; is to attach a debug container alongside the existing application container while keeping the original image unchanged:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;kubectl debug my-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=busybox \
  --share-processes \
  --container=debugger&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;This creates the copy-pod with both the original containers and a new &lt;code&gt;debugger&lt;/code&gt; container sharing the process namespace. Unlike ephemeral containers, this approach supports volume mounts and resource limits, and the debug pod can be deleted cleanly when you are done.&lt;/p&gt;



&lt;h3 class=&quot;wp-block-heading&quot;&gt;Limitations of the Copy Strategy&lt;/h3&gt;



&lt;p&gt;The pod copy approach has a critical limitation: &lt;strong&gt;it is not debugging the original pod&lt;/strong&gt;. It creates a new pod that may behave differently because:&lt;/p&gt;



&lt;ul class=&quot;wp-block-list&quot;&gt;
&lt;li&gt;It does not share the original pod&amp;rsquo;s &lt;strong&gt;in-memory state&lt;/strong&gt; &amp;mdash; if the issue is a goroutine leak or heap corruption that has been accumulating for hours, the fresh copy will not exhibit it immediately&lt;/li&gt;
&lt;li&gt;It creates a new Pod UID, which means any admission webhooks, network policies, or pod-level security contexts that depend on pod identity may apply differently&lt;/li&gt;
&lt;li&gt;If the original pod is crashing (&lt;code&gt;CrashLoopBackOff&lt;/code&gt;), the copy will also crash &amp;mdash; this technique does not help for crash debugging unless you also change the entrypoint&lt;/li&gt;
&lt;/ul&gt;



&lt;p&gt;For crash debugging specifically, combine &lt;code&gt;--copy-to&lt;/code&gt; with a modified entrypoint to keep the container alive:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;kubectl debug my-crashing-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=busybox \
  --share-processes \
  -- sleep 3600&lt;/code&gt;&lt;/pre&gt;



&lt;h2 class=&quot;wp-block-heading&quot;&gt;Option 3: Debug Image Variants&lt;/h2&gt;



&lt;p&gt;The most pragmatic approach &amp;mdash; and the one most appropriate for developer workflows &amp;mdash; is to maintain a debug variant of your application image that includes shell tooling. Both the Google distroless project and Chainguard provide this pattern officially.&lt;/p&gt;



&lt;p&gt;Google distroless images have a &lt;code&gt;:debug&lt;/code&gt; tag that adds BusyBox to the image:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;# Production image
FROM gcr.io/distroless/java17-debian12

# Debug variant &amp;mdash; identical but with BusyBox shell
FROM gcr.io/distroless/java17-debian12:debug&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;Chainguard images follow a similar convention with &lt;code&gt;:latest-dev&lt;/code&gt; variants that include apk, a shell, and common utilities:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;# Production (zero shell, minimal footprint)
FROM cgr.dev/chainguard/go:latest

# Development/debug variant
FROM cgr.dev/chainguard/go:latest-dev&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;If you build your own base images, the recommended approach is to use multi-stage builds and maintain separate build targets:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;FROM golang:1.22 AS builder
WORKDIR /app
COPY . .
RUN go build -o myapp .

# Production: static distroless image
FROM gcr.io/distroless/static-debian12 AS production
COPY --from=builder /app/myapp /myapp
ENTRYPOINT [&quot;/myapp&quot;]

# Debug variant: same binary, with shell tools
FROM gcr.io/distroless/static-debian12:debug AS debug
COPY --from=builder /app/myapp /myapp
ENTRYPOINT [&quot;/myapp&quot;]&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;In your CI/CD pipeline, build both targets and push &lt;code&gt;my-app:${VERSION}&lt;/code&gt; (production) and &lt;code&gt;my-app:${VERSION}-debug&lt;/code&gt; (debug variant) to your registry. The debug image is never deployed to production by default, but it exists and is ready to be used with &lt;code&gt;kubectl debug --copy-to&lt;/code&gt; when needed.&lt;/p&gt;



&lt;h3 class=&quot;wp-block-heading&quot;&gt;Security Considerations for Debug Variants&lt;/h3&gt;



&lt;p&gt;Debug image variants defeat much of the security benefit of distroless if they are used in production, even temporarily. Track usage carefully: log when debug images are deployed, require explicit approval, and ensure they are removed after the debugging session. In regulated environments, consider whether deploying a debug variant to production namespaces is permitted by your security policy &amp;mdash; in many cases it is not, and you must use ephemeral containers (which add a debug process to the pod without modifying the application image) instead.&lt;/p&gt;



&lt;h2 class=&quot;wp-block-heading&quot;&gt;Option 4: cdebug&lt;/h2&gt;



&lt;p&gt;&lt;code&gt;cdebug&lt;/code&gt; is an open-source CLI tool that simplifies distroless debugging by wrapping &lt;code&gt;kubectl debug&lt;/code&gt; with more ergonomic defaults and additional capabilities. Its primary value is in making ephemeral container debugging feel like a native shell experience:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;# Install
brew install cdebug
# or: go install github.com/iximiuz/cdebug@latest

# Debug a running pod
cdebug exec -it my-pod

# Specify a namespace and container
cdebug exec -it -n production my-pod -c my-container

# Use a specific debug image
cdebug exec -it my-pod --image=nicolaka/netshoot&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;What &lt;code&gt;cdebug&lt;/code&gt; adds over raw &lt;code&gt;kubectl debug&lt;/code&gt;:&lt;/p&gt;



&lt;ul class=&quot;wp-block-list&quot;&gt;
&lt;li&gt;&lt;strong&gt;Automatic filesystem chroot.&lt;/strong&gt; &lt;code&gt;cdebug exec&lt;/code&gt; automatically sets the filesystem root of the debug container to the target container&amp;rsquo;s filesystem, so you browse &lt;code&gt;/&lt;/code&gt; and see the application&amp;rsquo;s files &amp;mdash; not the debug image&amp;rsquo;s files. This addresses the most common friction point with &lt;code&gt;kubectl debug&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Docker integration.&lt;/strong&gt; &lt;code&gt;cdebug exec&lt;/code&gt; works identically for Docker containers (&lt;code&gt;cdebug exec -it &amp;lt;container-id&amp;gt;&lt;/code&gt;), making it the same muscle memory for local and cluster debugging.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No RBAC complications&lt;/strong&gt; for Docker-based local development &amp;mdash; useful for developer workflows before the code reaches Kubernetes.&lt;/li&gt;
&lt;/ul&gt;



&lt;p&gt;The tradeoff: &lt;code&gt;cdebug&lt;/code&gt; is a third-party dependency and requires installation. In environments with strict tooling policies (regulated industries, air-gapped clusters), it may not be an option. In those cases, the raw &lt;code&gt;kubectl debug&lt;/code&gt; workflow with &lt;code&gt;/proc/1/root&lt;/code&gt; filesystem navigation is the baseline.&lt;/p&gt;



&lt;h2 class=&quot;wp-block-heading&quot;&gt;Option 5: Node-Level Debugging&lt;/h2&gt;



&lt;p&gt;When everything else fails &amp;mdash; the pod is in &lt;code&gt;CrashLoopBackOff&lt;/code&gt; too fast to attach to, the issue is a kernel-level problem, or you need tools like &lt;code&gt;strace&lt;/code&gt; that require elevated privileges &amp;mdash; node-level debugging gives you direct access to the container&amp;rsquo;s processes from the host node.&lt;/p&gt;



&lt;p&gt;&lt;code&gt;kubectl debug node/&lt;/code&gt; creates a privileged pod on the target node that mounts the node&amp;rsquo;s root filesystem under &lt;code&gt;/host&lt;/code&gt;:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;kubectl debug node/my-node-name \
  -it \
  --image=nicolaka/netshoot&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;From this privileged pod, you can use &lt;code&gt;nsenter&lt;/code&gt; to enter the namespaces of any container running on the node:&lt;/p&gt;



&lt;pre class=&quot;wp-block-code&quot;&gt;&lt;code&gt;# Find the container's PID on the node
# (from within the node debug pod)
crictl ps | grep my-container
crictl inspect &amp;lt;container-id&amp;gt; | grep pid

#&lt;/code&gt;&lt;/pre&gt;
</description>
<category>Other</category>
<guid isPermaLink="true">https://coderlegion.com/15642/debugging-distroless-containers-kubectl-debug-ephemeral-containers-and-when-to-use-each</guid>
<pubDate>Fri, 24 Apr 2026 21:16:04 +0000</pubDate>
</item>
<item>
<title>ERROR (expected nested-name-specifier before ‘namespace’)-Fixed</title>
<link>https://coderlegion.com/13667/error-expected-nested-name-specifier-before-namespace-fixed</link>
<description>&lt;p&gt;The &lt;strong&gt;'ERROR (expected nested-name-specifier before ‘namespace’)`&lt;/strong&gt; error you may get while compiling your &lt;strong&gt;C++&lt;/strong&gt; code with a C++ compiler, this kind of issue might frustrate one, here we will discuss in the depth about the &lt;strong&gt;issue&lt;/strong&gt;. &lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://kodlogs.net/?qa=blob&amp;amp;qa_blobid=1477928991728469158&quot; alt=&quot;&quot; width=&quot;539&quot; height=&quot;359&quot;&gt;&lt;/p&gt;
&lt;h5 class=&quot;pk-title pk-toc-title pk-font-block&quot;&gt;Photo by &lt;a href=&quot;https://www.pexels.com/photo/people-standing-on-top-of-mountain-2114206/&quot; rel=&quot;nofollow&quot;&gt;Rachel Vine&lt;/a&gt;&lt;/h5&gt;
&lt;h5 class=&quot;pk-title pk-toc-title pk-font-block&quot;&gt;TABLE OF CONTENTS&lt;/h5&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a rel=&quot;nofollow&quot; href=&quot;#what-is-attributeerror-module-object-has-no-attribute-strptime&quot;&gt;What is &quot;ERROR (expected nested-name-specifier before ‘namespace’)&quot;?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;nofollow&quot; href=&quot;#Why-does-blocked-a-frame-with-origin-null-from-accessing-a-cross-origin-frame-happens&quot;&gt;Why &quot;&lt;/a&gt;&lt;a rel=&quot;nofollow&quot; href=&quot;#what-is-attributeerror-module-object-has-no-attribute-strptime&quot;&gt;ERROR (expected nested-name-specifier before ‘namespace’)&lt;/a&gt;&lt;a rel=&quot;nofollow&quot; href=&quot;#Why-does-blocked-a-frame-with-origin-null-from-accessing-a-cross-origin-frame-happens&quot;&gt;&quot; happens?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;nofollow&quot; href=&quot;#how-to-resolve&quot;&gt;How to resolve &quot;&lt;/a&gt;&lt;a rel=&quot;nofollow&quot; href=&quot;#what-is-attributeerror-module-object-has-no-attribute-strptime&quot;&gt;ERROR (expected nested-name-specifier before ‘namespace’)&lt;/a&gt;&lt;a rel=&quot;nofollow&quot; href=&quot;#how-to-resolve&quot;&gt;&lt;strong&gt;&quot;&lt;/strong&gt;:&lt;/a&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a rel=&quot;nofollow&quot; href=&quot;#solution-use-local-web-server&quot;&gt;Solution:&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;nofollow&quot; href=&quot;#conclusion&quot;&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;what-is-attributeerror-module-object-has-no-attribute-strptime&quot;&gt;&lt;strong&gt;What is  “ERROR (expected nested-name-specifier before ‘namespace’)&lt;/strong&gt;&lt;strong&gt;”?&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;Before discussing the error, we should know, what is a &lt;strong&gt;&lt;code&gt;nested name specifier&lt;/code&gt;&lt;/strong&gt;?&lt;/p&gt;
&lt;p&gt;C++ nested name specifiers are the &lt;strong&gt;prefixes &lt;/strong&gt;to qualified names. For example,&lt;code&gt; “foo::” in “foo::x”&lt;/code&gt; is a nested name specifier. Nested name specifiers are made up of a sequence of specifiers, each of which can be namespace, type, identifier (for dependent names), &lt;strong&gt;decltype specifier&lt;/strong&gt;, or the &lt;strong&gt;global specifier (‘::’)&lt;/strong&gt;. The last two specifiers can only appear at the start of a nested-namespace-specifier.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;#include &lt;/code&gt;&lt;br&gt;&lt;code&gt;#include &lt;/code&gt;&lt;/p&gt;
&lt;p&gt;&lt;code&gt;template&lt;/code&gt;&lt;br&gt;&lt;code&gt;class ListPtr;&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;&lt;code&gt;template&lt;/code&gt;&lt;br&gt;&lt;code&gt;class ListPtrIterator {&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;&lt;code&gt;public:&lt;/code&gt;&lt;br&gt;&lt;code&gt;ListPtrIterator( ListPtr* _user ):&lt;/code&gt;&lt;br&gt;&lt;code&gt;M_user( _user ),&lt;/code&gt;&lt;br&gt;&lt;code&gt;M_cur( M_user-&amp;gt;begin() ) {}&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;&lt;code&gt;bool&lt;/code&gt;&lt;br&gt;&lt;code&gt;cur( T*&amp;amp; );&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;&lt;code&gt;bool&lt;/code&gt;&lt;br&gt;&lt;code&gt;reset( void ) {&lt;/code&gt;&lt;br&gt;&lt;code&gt;M_cur =M_user-&amp;gt;begin();&lt;/code&gt;&lt;br&gt;&lt;code&gt;return M_cur!=M_user-&amp;gt;end();&lt;/code&gt;&lt;br&gt;&lt;code&gt;}&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;&lt;code&gt;private:&lt;/code&gt;&lt;br&gt;&lt;code&gt;ListPtr* M_user;&lt;/code&gt;&lt;br&gt;&lt;code&gt;typename list::iterator M_cur;&lt;/code&gt;&lt;br&gt;&lt;code&gt;^^^^&lt;/code&gt;&lt;br&gt;&lt;code&gt;ERROR: expected nested-name-specifier before 'list'&lt;/code&gt;&lt;br&gt;&lt;code&gt;};&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Output&lt;/strong&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;new.cpp:28:46: warning: multi-character character constant [-Wmultichar]

 ERROR: expected nested-name-specifier before 'list'&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&quot;Why-does-blocked-a-frame-with-origin-null-from-accessing-a-cross-origin-frame-happens&quot;&gt;&lt;strong&gt;Why does “ERROR (expected nested-name-specifier before ‘namespace’)&lt;/strong&gt;&lt;strong&gt;“ happens?&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;As the compiler suggests that there is an with the list. &lt;strong&gt;Here the compiler does not accept the list as it is&lt;/strong&gt;. Compiler accepts it as std::list.&lt;/p&gt;
&lt;p&gt;As we know if we do not use the using namespace std in our program then we will face an error saying may be like that cout was not declared in this scope. But we know that in the past we did not need to define the using namespace std in the program. But now we have to use it or we have to use cout as std::cout. &lt;/p&gt;
&lt;p&gt;In the program given above the cause is pretty similar to this reason.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Following is the error message you get:&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;code&gt;` main.cpp online 2:7: error: expected nested-name-specifier before ‘namespace’&lt;br&gt; using namespace std;&lt;br&gt;       ^~~~~~~~~`&lt;/code&gt;&lt;/p&gt;
&lt;p&gt; &lt;/p&gt;
&lt;h2&gt;&lt;strong&gt;How to resolve “ERROR (expected nested-name-specifier before ‘namespace’)&lt;/strong&gt;&lt;strong&gt;”?&lt;/strong&gt;&lt;/h2&gt;
&lt;h2 id=&quot;solution-1-use-local-web-server&quot; style=&quot;line-height: 1;&quot;&gt;&lt;strong&gt;Solution :&lt;/strong&gt;&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;As I mentioned earlier, in the program given above the cause is pretty similar to this reason.&lt;/strong&gt; So we have to use std::list to resolve the error since compilers do not accept containers without nested-name-specifier.&lt;/p&gt;
&lt;p&gt; &lt;/p&gt;
&lt;p style=&quot;line-height: 1;&quot;&gt;⚠️ Remember to use C++11 (otherwise you'll receive an error saying that extended initializer lists are only available with -std=c++11 or -std=gnu++11)..&lt;/p&gt;
&lt;p style=&quot;line-height: 1;&quot;&gt; &lt;/p&gt;
&lt;p&gt;ℹ   Ensure that the relevant header files are included.&lt;/p&gt;
&lt;h3 id=&quot;conclusion&quot; style=&quot;text-align: left;&quot;&gt;&lt;strong&gt;Conclusion:&lt;/strong&gt;&lt;/h3&gt;
&lt;p style=&quot;text-align: left;&quot;&gt; We discussed in this article about &lt;strong&gt;ERROR (expected nested-name-specifier before ‘namespace’) discussed &lt;/strong&gt;and a number of &lt;strong&gt;solutions &lt;/strong&gt;to get around it, one final way out may be be to upgrade to a more recent compiler version &lt;/p&gt;
</description>
<category>Articles</category>
<guid isPermaLink="true">https://coderlegion.com/13667/error-expected-nested-name-specifier-before-namespace-fixed</guid>
<pubDate>Mon, 30 Mar 2026 03:20:18 +0000</pubDate>
</item>
</channel>
</rss>