Skip to content

lightning OOM during write/ingest to TiKV when import large mount of data #59947

Closed
#60097
Closed
lightning OOM during write/ingest to TiKV when import large mount of data#59947
#60097
Labels
affects-8.1This bug affects the 8.1.x(LTS) versions.affects-8.5This bug affects the 8.5.x(LTS) versions.component/lightningThis issue is related to Lightning of TiDB.impact/oomimpact/upgradeseverity/majortype/bugThe issue is confirmed as a bug.type/regression

Description

@D3Hunter
D3Hunter
opened on Mar 7, 2025

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

the issue happened in two cases, one 2TB data, another 12TB data, both are using v8.1.2 version. current guess of the root cause is that pebble might have some bug, causes memory usage burst, and OOM, as the time window is very short, and pebble might allocate memory using CGO, we hasn't got any useful heap profile.
and we found that in v8.1.1 we have upgrade pebble to v1.1 in 888a58b since v8.1.1. so we try import the same data using v7.5.5, and it success.

we found a pebble OOM issue cockroachdb/pebble#3039, it might related

so the trigger condition might include

  • import large data set, say 2T. (we have tested with smaller data-set, say several hundreds of GB, no such issue)
  • using version >= v8.1.1 which have upgraded pebble

2. What did you expect to see? (Required)

success

3. What did you see instead (Required)

oom on ingest

4. What is your TiDB version? (Required)

v8.1.2

Activity

  1. D3Hunter
    added
    affects-8.1This bug affects the 8.1.x(LTS) versions.
    affects-8.5This bug affects the 8.5.x(LTS) versions.
    component/lightningThis issue is related to Lightning of TiDB.
    severity/critical
    type/bugThe issue is confirmed as a bug.
    on Mar 7, 2025
  2. ti-chi-bot
    added
    may-affects-6.1
    may-affects-6.5
    may-affects-7.1
    may-affects-7.5
    on Mar 7, 2025
  3. D3Hunter
    added
    severity/major
    and removed
    severity/critical
    on Mar 7, 2025
  4. jebter
    added
    impact/oom
    on Mar 7, 2025

  5. kennedy8312 commented on Mar 13, 2025

    @kennedy8312
    kennedy8312
    on Mar 13, 2025

    /type regression
    Mark this one as regression since the same operation succeeded on an earlier version.

  6. ti-chi-bot
    added
    type/regression
    on Mar 13, 2025

  7. OliverS929 commented on Mar 16, 2025

    @OliverS929
    OliverS929
    on Mar 16, 2025
    Contributor

    /remove may-affects-6.1

  8. 4 remaining items

  9. ti-chi-bot
    removed
    may-affects-6.5
    on Mar 17, 2025

  10. OliverS929 commented on Mar 17, 2025

    @OliverS929
    OliverS929
    on Mar 17, 2025
    Contributor

    /remove-label may-affects-7.1

  11. ti-chi-bot
    removed
    may-affects-7.1
    on Mar 17, 2025

  12. OliverS929 commented on Mar 17, 2025

    @OliverS929
    OliverS929
    on Mar 17, 2025
    Contributor

    /remove-label may-affects-7.5

  13. ti-chi-bot
    removed
    may-affects-7.5
    on Mar 17, 2025
  14. ti-chi-bot
    closed this as completedin #60097on Mar 20, 2025
  15. ti-chi-bot
    closed this as completedin 514204eon Mar 20, 2025
  16. qw4990
    added a commit that references this issue on Mar 20, 2025

    Lightning: Make sure we are using default block size of 16KB if user …

    514204e
  17. ti-chi-bot
    added 2 commits that reference this issue on Mar 24, 2025

    Lightning: Make sure we are using default block size of 16KB if user …

    9679a3f

    Lightning: Make sure we are using default block size of 16KB if user …

    a6eeab5
  18. disksing
    added a commit that references this issue on Mar 27, 2025

    Lightning: Make sure we are using default block size of 16KB if user …

    8c81e79
  19. seiya-annie
    added
    impact/upgrade
    on Apr 11, 2025

  20. seiya-annie commented on Apr 11, 2025

    @seiya-annie
    seiya-annie
    on Apr 11, 2025

    Upgrade branch meet this issue:
    source: version <7.6.0
    target: version >=7.6.0
    blocking stage: post upgrade
    workaround: downgrade

  21. zeminzhou
    added a commit that references this issue on May 6, 2025

    Lightning: Make sure we are using default block size of 16KB if user …

    9ae3b30

  22. jebter commented on May 29, 2025

    @jebter
    jebter
    on May 29, 2025

    Regression Analysis
    PR caused this regression: #49514
    Related to issue/feature: #45037

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-8.1This bug affects the 8.1.x(LTS) versions.affects-8.5This bug affects the 8.5.x(LTS) versions.component/lightningThis issue is related to Lightning of TiDB.impact/oomimpact/upgradeseverity/majortype/bugThe issue is confirmed as a bug.type/regression

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions