Skip to content

[tidb-x] data inconsistency in table after simulated kill tikv-worker repeatedly #64666

Closed

Description

@Lily2025

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run sysbench
2、add index for two tables
3、simulated kill all tikv-worker repeatedly

  • name: "ha_tikv_worker_all_kill"
    warmUpTime: 1m
    faultType: kill
    selector: "workloadnode-tikv-worker(all)"
    period: "@every 3m"
    faultDuration: 1m
    faultTotalRunTime: 30m
    ddl:
    addIndex:
    - db: "partitioninsert_1647963896275"
    table: "sbtest1"
    timeout: 60m
    times: 7
    index_col_name: "c"
    job_type: "ingest"
    admin_check: true
    - db: "partitioninsert_1647963896275"
    table: "sbtest2"
    timeout: 60m
    times: 7
    index_col_name: "c"
    job_type: "ingest"
    admin_check: true

2. What did you expect to see? (Required)

data inconsistency in table if add index successfully

3. What did you see instead (Required)

data inconsistency in table after simulated kill tidb-worker repeatedly

mysql> admin show ddl jobs;
+--------+---------------------------------------------------+------------+-------------+----------------------+-----------+----------+-----------+----------------------------+----------------------------+----------------------------+---------------+----------+
| JOB_ID | DB_NAME | TABLE_NAME | JOB_TYPE | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | CREATE_TIME | START_TIME | END_TIME | STATE | COMMENTS |
+--------+---------------------------------------------------+------------+-------------+----------------------+-----------+----------+-----------+----------------------------+----------------------------+----------------------------+---------------+----------+
| 7408 | partitioninsert_1647963896275 | sbtest2 | add index | public | 4144 | 3889 | 0 | 2025-11-25 06:29:25.478000 | 2025-11-25 06:29:25.478000 | 2025-11-25 07:22:51.128000 | synced | |

case logs
admin check failed (Error 8223 (HY000): data inconsistency in table: sbtest2, index: index_test_1764052165225, handle: 5402730, index-values:"" != record-values:"handle: 5402730, values: [KindString 16960352122-53417868865-19761453644-60931175938-11266666891-26482692461-21719450064-35170464548-23376004353-52869623519]")
operatorLogs:
[2025-11-25 14:29:25] ###### start adding index
ALTER TABLE sbtest2 ADD INDEX index_test_1764052165225(c)
[2025-11-25 14:29:25] ###### wait for ddl job finish
[2025-11-25 15:22:53] ###### ddl job finished
select job_id, db_name, table_name, job_type, create_time, start_time, end_time, state, query from information_schema.ddl_jobs where query = 'ALTER TABLE sbtest2 ADD INDEX index_test_1764052165225(c)'
jobId: 7408, job type: add index, state: synced, comments:
add index done, it takes: 53m27.992383292s
[2025-11-25 15:22:53] ###### start admin check

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v9.0.0-beta.2.pre-810-gbf57e58
Edition: Enterprise
Git Commit Hash: bf57e58
Git Branch: HEAD
UTC Build Time: 2025-11-24 03:31:04
GoVersion: go1.23.12
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Enterprise Extension Commit Hash: dc633aae52eb11b4e3549e3a7ef5b0ed14e159b1
Kernel Type: Next Generation

Activity

  1. Lily2025 commented on Nov 25, 2025

    @Lily2025
    Author

    /severity critical
    /assign gmhdbjd

  2. changed the title [-][next-gen] data inconsistency in table after simulated kill tidb-worker repeatedly[/-] [+][next-gen] data inconsistency in table after simulated kill tikv-worker repeatedly[/+] on Dec 1, 2025
  3. Lily2025 commented on Dec 1, 2025

    @Lily2025
    Author

    /assign wjhuang2016

  4. changed the title [-][next-gen] data inconsistency in table after simulated kill tikv-worker repeatedly[/-] [+][tidb-x] data inconsistency in table after simulated kill tikv-worker repeatedly[/+] on Dec 5, 2025
  5. added 2 commits that reference this issue on Dec 14, 2025
    5810fff
    3ae9469
  6. added a commit that references this issue on Apr 21, 2026
    f83e2fd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

affects-8.1This bug affects the 8.1.x(LTS) versions.affects-8.5This bug affects the 8.5.x(LTS) versions.component/DXFDistributed eXecute Framework of tidbcomponent/ddlThis issue is related to DDL of TiDB.severity/criticaltype/bugThe issue is confirmed as a bug.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions