Skip to content

doesn't detect hubble #39

Closed
#49
Closed
doesn't detect hubble#39
#49
Labels
bugSomething isn't working

Description

@fenio
fenio
opened on Jan 30, 2026

Describe the bug

Radar can't find my hubble install.

To reproduce

Steps to reproduce the behavior:

  1. launch radar
  2. Click on 'Traffic'
  3. See error

Expected behavior

Working traffic visualization.

Screenshots

Image

Environment

  • Kubernetes version: 1.35.0
  • Cluster type: Talos

Logs

2026/01/30 16:53:36 [caretta] Namespace caretta not found, checking default namespace
2026/01/30 16:53:36 "GET http://localhost:9280/api/traffic/sources HTTP/1.1" from 127.0.0.1:63858 - 200 258B in 176.543208ms
2026/01/30 16:53:36 "POST http://localhost:9280/api/traffic/connect HTTP/1.1" from 127.0.0.1:63858 - 200 19B in 241.875Âĩs
2026/01/30 16:53:36 [hubble] HTTP API not available: Get "http://hubble-ui.kube-system.svc.cluster.local:12000/api/v1/flows": dial tcp: lookup hubble-ui.kube-system.svc.cluster.local: no such host

Additional context

[â˜ļ omni-homelab:kube-system] [⏱ 2s]
âŊ ~ kg svc
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
cilium-agent      ClusterIP   None            <none>        9962/TCP                 70d
cilium-envoy      ClusterIP   None            <none>        9964/TCP                 70d
cilium-operator   ClusterIP   None            <none>        9963/TCP                 70d
hubble-metrics    ClusterIP   None            <none>        9965/TCP                 70d
hubble-peer       ClusterIP   10.104.82.241   <none>        443/TCP                  70d
hubble-relay      ClusterIP   10.96.164.66    <none>        80/TCP                   70d
hubble-ui         ClusterIP   10.103.5.187    <none>        80/TCP                   70d
kube-dns          ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP   70d
tns-csi-metrics   ClusterIP   10.100.155.47   <none>        8080/TCP                 12d
vm-core-dns       ClusterIP   None            <none>        9153/TCP                 69d

Activity

  1. fenio
    added
    bugSomething isn't working
    on Jan 30, 2026

  2. nadaverell commented on Jan 30, 2026

    @nadaverell
    nadaverell
    on Jan 30, 2026
    Contributor

    Fixed in #42 - Hubble detection now works correctly and connects via gRPC to Hubble Relay.

  3. nadaverell
    closed this as completedon Jan 30, 2026

  4. fenio commented on Jan 31, 2026

    @fenio
    fenio
    on Jan 31, 2026
    Author

    Well it still doesn't work for me.

    That's relevant part of values I'm using for hubble-ui:

      ui:
    enabled: true
    rollOutPods: true
    backend:
      image:
        pullPolicy: Always
      resources:
        requests:
          cpu: 10m
          memory: 100Mi
          ephemeral-storage: 100Mi
        limits:
          memory: 200Mi
          ephemeral-storage: 200Mi
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities: {drop: ["ALL"]}
    frontend:
      image:
        pullPolicy: Always
      resources:
        requests:
          cpu: 10m
          memory: 100Mi
          ephemeral-storage: 100Mi
        limits:
          memory: 200Mi
          ephemeral-storage: 200Mi
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities: {drop: ["ALL"]}

  5. nadaverell commented on Jan 31, 2026

    @nadaverell
    nadaverell
    on Jan 31, 2026
    Contributor

    Thanks, I'll try to recreate with your configuration

  6. nadaverell commented on Jan 31, 2026

    @nadaverell
    nadaverell
    on Jan 31, 2026
    Contributor

    @fenio can you share a bit more about your setup? Cloud provider? Can you check where hubble-relay (not UI) is deployed and show me its configuration?

    looking now I think in GKE by default it uses a different namespace and also requires TLS to access the hubble-relay endpoint.

  7. nadaverell
    added a commit that references this issue on Jan 31, 2026

    Fix Hubble detection for GKE Dataplane V2 and add TLS support

    83e1344

  8. nadaverell commented on Jan 31, 2026

    @nadaverell
    nadaverell
    on Jan 31, 2026
    Contributor

    This has been fully fixed in commit 83e1344.

    Changes:

    • Switched from hardcoded kube-system namespace to label-based discovery (k8s-app=hubble-relay) across all namespaces
    • Added TLS support with automatic certificate loading from hubble-relay-client-certs secret
    • Fixed ServerName in TLS config for port-forward scenarios
    • Added scoped RBAC in Helm chart for the TLS cert secret

    Tested on:

    • GKE Autopilot with Dataplane V2 (namespace: gke-managed-dpv2-observability, TLS required)
    • EKS with Cilium (namespace: kube-system, no TLS)

    The fix should also work for standard Cilium installations like Talos where Hubble Relay is in kube-system.

  9. nadaverell commented on Jan 31, 2026

    @nadaverell
    nadaverell
    on Jan 31, 2026
    Contributor

    @fenio test latest release and let me know if this works now!

  10. fenio commented on Jan 31, 2026

    @fenio
    fenio
    on Jan 31, 2026
    Author

    Works! Thank you.

  11. fenio commented on Feb 1, 2026

    @fenio
    fenio
    on Feb 1, 2026
    Author

    Well worked in 0.7.0. In 0.7.3 it fails again. I mean it detects hubble-ui at least but it's having some weird issues connecting to it.

    2026/02/01 15:02:57 [hubble] Starting port-forward to kube-system/hubble-relay:4245
2026/02/01 15:02:58 [traffic] Metrics port-forward ready: localhost:56005 -> kube-system/hubble-relay:4245 (context: omni-homelab)
2026/02/01 15:02:58 [hubble] Connecting to gRPC at localhost:56005 (TLS: true)
E0201 15:02:58.178187   89987 portforward.go:424] "Unhandled Error" err="an error occurred forwarding 56005 -> 4245: error forwarding port 4245 to pod a68b8d206d91bcc8e11e2e4dcc0a7b848cd1f2f801702480c3ed53ed180f3522, uid : failed to execute portforward in network namespace \"/var/run/netns/cni-9930a732-2877-e381-f451-c1f54a45f0ec\": writeto tcp4 127.0.0.1:53926->127.0.0.1:4245: read tcp4 127.0.0.1:53926->127.0.0.1:4245: read: connection reset by peer" logger="UnhandledError"
E0201 15:02:58.178187   89987 portforward.go:424] "Unhandled Error" err="an error occurred forwarding 56005 -> 4245: error forwarding port 4245 to pod a68b8d206d91bcc8e11e2e4dcc0a7b848cd1f2f801702480c3ed53ed180f3522, uid : failed to execute portforward in network namespace \"/var/run/netns/cni-9930a732-2877-e381-f451-c1f54a45f0ec\": writeto tcp4 127.0.0.1:53926->127.0.0.1:4245: read tcp4 127.0.0.1:53926->127.0.0.1:4245: read: connection reset by peer" logger="UnhandledError"
E0201 15:02:58.178187   89987 portforward.go:424] "Unhandled Error" err="an error occurred forwarding 56005 -> 4245: error forwarding port 4245 to pod a68b8d206d91bcc8e11e2e4dcc0a7b848cd1f2f801702480c3ed53ed180f3522, uid : failed to execute portforward in network namespace \"/var/run/netns/cni-9930a732-2877-e381-f451-c1f54a45f0ec\": writeto tcp4 127.0.0.1:53926->127.0.0.1:4245: read tcp4 127.0.0.1:53926->127.0.0.1:4245: read: connection reset by peer" logger="UnhandledError"
E0201 15:02:58.178187   89987 portforward.go:424] "Unhandled Error" err="an error occurred forwarding 56005 -> 4245: error forwarding port 4245 to pod a68b8d206d91bcc8e11e2e4dcc0a7b848cd1f2f801702480c3ed53ed180f3522, uid : failed to execute portforward in network namespace \"/var/run/netns/cni-9930a732-2877-e381-f451-c1f54a45f0ec\": writeto tcp4 127.0.0.1:53926->127.0.0.1:4245: read tcp4 127.0.0.1:53926->127.0.0.1:4245: read: connection reset by peer" logger="UnhandledError"
E0201 15:02:58.187187   89987 portforward.go:404] "Unhandled Error" err="error copying from local connection to remote stream: EOF" logger="UnhandledError"
E0201 15:02:58.187187   89987 portforward.go:404] "Unhandled Error" err="error copying from local connection to remote stream: EOF" logger="UnhandledError"
E0201 15:02:58.187187   89987 portforward.go:404] "Unhandled Error" err="error copying from local connection to remote stream: EOF" logger="UnhandledError"
E0201 15:02:58.187187   89987 portforward.go:404] "Unhandled Error" err="error copying from local connection to remote stream: EOF" logger="UnhandledError"
2026/02/01 15:02:58 [hubble] Connection test failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: EOF"
2026/02/01 15:02:58 "POST http://localhost:9280/api/traffic/connect HTTP/1.1" from 127.0.0.1:55996 - 200 77B in 624.336208ms
2026/02/01 15:02:58 "GET http://localhost:9280/api/traffic/flows?since=5m HTTP/1.1" from 127.0.0.1:55995 - 200 207B in 625.094292ms
  12. nadaverell
    added a commit that references this issue on Feb 1, 2026

    fix: Use service port heuristic for Hubble TLS detection

    0ae73e8

  13. nadaverell commented on Feb 1, 2026

    @nadaverell
    nadaverell
    on Feb 1, 2026
    Contributor

    It should use hubble-relay, not UI.
    I think the issue might be that instead of optionally using TLS, we now force it.
    Attempting a fix

  14. nadaverell commented on Feb 1, 2026

    @nadaverell
    nadaverell
    on Feb 1, 2026
    Contributor

    @fenio Can you try again with latest release?

  15. fenio commented on Feb 1, 2026

    @fenio
    fenio
    on Feb 1, 2026
    Author

    Works again ;)

  16. nadaverell commented on Feb 1, 2026

    @nadaverell
    nadaverell
    on Feb 1, 2026
    Contributor

    Great! Thanks for reporting and for your patience :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions