Repository navigation
global sort importinto stuck at ingest step due to memory limiter #62388
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
this case happens when we try to import a 7.5G table using 2 threads on a 8c16g machine which only have 12.9G memory usable, so for a task of thread=2, it can only use 3.225GiB, and of which 1.488GiB can be used to load ingested kv into memory
the issue happens in nextgen kernel, but it's a issue of global-sort, so also affects classic kernel
the stuck happens when ingesting index kv, the reason is:
- each index kv is only 42B, but to store it in memory we need additional 48 for slice header, so it nearly double the memory
- concurrent file read also use the same memory limiter and need to read all related data
- add up all above 2, it exceed the memory limiter size which is 1.488GiB
in previous version, before #58572, we need at least 8 thread to run importinto with global sort, so it only affects master
goroutine 7636944 [chan receive, 2679 minutes]:
github.com/pingcap/tidb/pkg/lightning/membuf.(*Limiter).Acquire(0xc00ec9ea50, 0x40000)
/workspace/source/tidb/pkg/lightning/membuf/limiter.go:56 +0x1ab
github.com/pingcap/tidb/pkg/lightning/membuf.(*Buffer).recordSmallObjOverhead(...)
/workspace/source/tidb/pkg/lightning/membuf/buffer.go:197
github.com/pingcap/tidb/pkg/lightning/membuf.(*Buffer).AllocBytes(0xc0012712c0, 0x29?)
/workspace/source/tidb/pkg/lightning/membuf/buffer.go:257 +0x6d
github.com/pingcap/tidb/pkg/lightning/membuf.(*Buffer).AddBytes(...)
/workspace/source/tidb/pkg/lightning/membuf/buffer.go:328
github.com/pingcap/tidb/pkg/lightning/backend/external.readOneFile({0x75a33b0, 0xc004bf5090}, {0x75ce0b0, 0xc064b04750}, {0xc013796280, 0x3a}, {0xc00a6e8720, 0x29, 0x2a}, {0xc00a6e8780, ...}, ...)
/workspace/source/tidb/pkg/lightning/backend/external/reader.go:184 +0x3c5
github.com/pingcap/tidb/pkg/lightning/backend/external.readAllData.func2()
/workspace/source/tidb/pkg/lightning/backend/external/reader.go:98 +0x3b8
github.com/pingcap/tidb/pkg/util.(*ErrorGroupWithRecover).Go.func1()
/workspace/source/tidb/pkg/util/wait_group_wrapper.go:263 +0x52
golang.org/x/sync/errgroup.(*Group).add.func1()
/root/go/pkg/mod/golang.org/x/sync@v0.15.0/errgroup/errgroup.go:128 +0x7e
created by golang.org/x/sync/errgroup.(*Group).add in goroutine 7636925
/root/go/pkg/mod/golang.org/x/sync@v0.15.0/errgroup/errgroup.go:96 +0x79
2. What did you expect to see? (Required)
3. What did you see instead (Required)
4. What is your TiDB version? (Required)
master
Activity
D3Hunter commentedon Jul 14, 2025
as add-index also use global sort, the default concurrency is 4, might meet this issue too, so also affects older version
globalsort: fix stuck when ingesting small kv (pingcap#62407)
globalsort: fix stuck when ingesting small kv (#62407) (#62409)
globalsort: fix stuck when ingesting small kv (#62407) (#62410)