Skip to content

Changing app-protocol from HTTP2 to other protocols will cause glbc to hit nil pointer  #675

Closed
Closed
Changing app-protocol from HTTP2 to other protocols will cause glbc to hit nil pointer #675
Assignees
freehan
Labels
kind/bugCategorizes issue or PR as related to a bug.lifecycle/frozenIndicates that an issue or PR should not be auto-closed due to staleness.

Description

@freehan
freehan
opened on Mar 11, 2019

Symptom: Create an ingress with HTTP2 app protocol. For instance: https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/ingress/http2
Manually change the app protocol on the spec of service from HTTP2 to something like HTTPS,HTTP. This will cause glbc to hit nil pointer and crash with following stacktrace in ingress-gce 1.2:

/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:573
/usr/local/go/src/runtime/panic.go:502
/usr/local/go/src/runtime/panic.go:63
/usr/local/go/src/runtime/signal_unix.go:388
/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:332
/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:244
/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:106
/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:220
/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:283
/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:252
/go/src/k8s.io/ingress-gce/pkg/controller/cluster_manager.go:98
/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:349
/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:299
/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:111
/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:84
/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:54
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:54
/usr/local/go/src/runtime/asm_amd64.s:2361
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1674e63]

goroutine 99 [running]:
k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x19a12a0, 0x29aa020)
	/usr/local/go/src/runtime/panic.go:502 +0x229
k8s.io/ingress-gce/pkg/healthchecks.NewHealthCheck(...)
	/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:332
k8s.io/ingress-gce/pkg/healthchecks.(*HealthChecks).Get(0xc42007bec0, 0xc420a8e3e0, 0x1e, 0x1cda028, 0x2, 0x0, 0x0, 0x0)
	/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:244 +0x313
k8s.io/ingress-gce/pkg/healthchecks.(*HealthChecks).Sync(0xc42007bec0, 0xc42068b880, 0xc, 0xc420518c10, 0xb, 0x0)
	/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:106 +0xae
k8s.io/ingress-gce/pkg/backends.(*Backends).ensureHealthCheck(0xc420303960, 0xc420518b90, 0xc, 0xc420518c10, 0xb, 0x0, 0x1bb, 0x0, 0x0, 0x7f21, ...)
	/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:220 +0xc0
k8s.io/ingress-gce/pkg/backends.(*Backends).ensureBackendService(0xc420303960, 0xc420518b90, 0xc, 0xc420518c10, 0xb, 0x0, 0x1bb, 0x0, 0x0, 0x7f21, ...)
	/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:283 +0x233
k8s.io/ingress-gce/pkg/backends.(*Backends).Ensure(0xc420303960, 0xc4208ab980, 0x1, 0x1, 0xc4203d1b30, 0x1, 0x1, 0x1670ad8, 0x184a100)
	/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:252 +0x1fd
k8s.io/ingress-gce/pkg/controller.(*ClusterManager).EnsureLoadBalancer(0xc4208f7680, 0xc4203f9b20, 0xc4208aa380, 0x1, 0x1, 0xc4203d1b30, 0x1, 0x1, 0x1, 0x1)
	/go/src/k8s.io/ingress-gce/pkg/controller/cluster_manager.go:98 +0x1ed
k8s.io/ingress-gce/pkg/controller.(*LoadBalancerController).ensureIngress(0xc4204405a0, 0xc4208b7a20, 0xc420771c00, 0x3, 0x4, 0xc4208aa280, 0x1, 0x1, 0x0, 0x0)
	/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:349 +0x4b6
k8s.io/ingress-gce/pkg/controller.(*LoadBalancerController).sync(0xc4204405a0, 0xc42039dd80, 0x14, 0xc420699b01, 0xc420472d00)
	/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:299 +0x3e5
k8s.io/ingress-gce/pkg/controller.(*LoadBalancerController).(k8s.io/ingress-gce/pkg/controller.sync)-fm(0xc42039dd80, 0x14, 0xf, 0xc4208dddd8)
	/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:111 +0x3e
k8s.io/ingress-gce/pkg/utils.(*PeriodicTaskQueue).worker(0xc420400240)
	/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:84 +0x17a
k8s.io/ingress-gce/pkg/utils.(*PeriodicTaskQueue).(k8s.io/ingress-gce/pkg/utils.worker)-fm()
	/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:54 +0x2a
k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc42072dfa8)
	/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc420a1ffa8, 0x3b9aca00, 0x0, 0x1, 0xc4200848a0)
	/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc42072dfa8, 0x3b9aca00, 0xc4200848a0)
	/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
k8s.io/ingress-gce/pkg/utils.(*PeriodicTaskQueue).Run(0xc420400240, 0x3b9aca00, 0xc4200848a0)
	/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:54 +0x55
created by k8s.io/ingress-gce/pkg/controller.(*LoadBalancerController).Run
	/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:231 +0x86

Root Cause:

  1. GCE GetHealthCheck V1 API used to return the health check type to "HTTP" while the healthcheck object is create beta API using "HTTP2".

  2. Now HTTP2 is promoted to GA on GCE. GCE GetHealthCheck V1 API returns health check type to "HTTP2".

  3. The GCE go client used by ingress-gce does not have the new HTTP2 fields in the V1 Healthcheck API.

  4. In case 1, this path is hit https://github.com/kubernetes/ingress-gce/blob/release-1.2/pkg/healthchecks/healthchecks.go#L328

  5. With 2, this path is hit https://github.com/kubernetes/ingress-gce/blob/release-1.2/pkg/healthchecks/healthchecks.go#L332 hc.Http2HealthCheck is nil because the V1 healthcheck object does not contain HTTP2HealthCheck fields due to 3

Impacted Versions: 1.2, 1.3, 1.4, 1.5, HEAD

Activity

  1. rramkumar1
    added
    kind/bugCategorizes issue or PR as related to a bug.
    on Mar 11, 2019
  2. rramkumar1
    assigned
    freehan
    on Mar 12, 2019

  3. fejta-bot commented on Jun 12, 2019

    @fejta-bot
    fejta-bot
    on Jun 12, 2019

    Issues go stale after 90d of inactivity.
    Mark the issue as fresh with /remove-lifecycle stale.
    Stale issues rot after an additional 30d of inactivity and eventually close.

    If this issue is safe to close now please do so with /close.

    Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
    /lifecycle stale

  4. k8s-ci-robot
    added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Jun 12, 2019

  5. bowei commented on Jun 12, 2019

    @bowei
    bowei
    on Jun 12, 2019
    Member

    /lifecycle frozen

  6. k8s-ci-robot
    added
    lifecycle/frozenIndicates that an issue or PR should not be auto-closed due to staleness.
    and removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Jun 12, 2019

  7. rramkumar1 commented on Jul 10, 2020

    @rramkumar1
    rramkumar1
    on Jul 10, 2020
    Contributor

    Closing since this issue is fully mitigated.

  8. rramkumar1
    closed this as completedon Jul 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind/bugCategorizes issue or PR as related to a bug.lifecycle/frozenIndicates that an issue or PR should not be auto-closed due to staleness.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions