Skip to content

TiDB：runtime error: invalid memory address or nil pointer dereference #65867

Closed
Closed
TiDB：runtime error: invalid memory address or nil pointer dereference#65867
Assignees
hawkingrei
Labels
affects-8.5This bug affects the 8.5.x(LTS) versions.contributionThis PR is from a community contributor.first-time-contributorIndicates that the PR was contributed by an external member and is a first-time contributor.impact/panicreport/customerCustomers have encountered this bug.severity/moderatesig/plannerSIG: Plannertype/bugThe issue is confirmed as a bug.

Description

@caiyifeng-1994
caiyifeng-1994
opened on Jan 28, 2026

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

This table always being ‘inset’and 'delete' , and suddenly the following error popped up:

mysql> select * from ACT_RU_VARIABLE where task_id_='1997872469436538880';
ERROR 1105 (HY000): runtime error: invalid memory address or nil pointer dereference

It recovers after doing the steps below, but the issue keeps coming back frequently.

DROP INDEX ACT_IDX_VARIABLE_TASK_ID ON ACT_RU_VARIABLE;
CREATE INDEX ACT_IDX_VARIABLE_TASK_ID ON ACT_RU_VARIABLE(TASK_ID_);

table info and field :

CREATE TABLE `ACT_RU_VARIABLE` (
  `ID_` varchar(64) NOT NULL,
  `REV_` int DEFAULT NULL,
  `TYPE_` varchar(255) NOT NULL,
  `NAME_` varchar(255) NOT NULL,
  `EXECUTION_ID_` varchar(64) DEFAULT NULL,
  `PROC_INST_ID_` varchar(64) DEFAULT NULL,
  `TASK_ID_` varchar(64) DEFAULT NULL,
  `SCOPE_ID_` varchar(255) DEFAULT NULL,
  `SUB_SCOPE_ID_` varchar(255) DEFAULT NULL,
  `SCOPE_TYPE_` varchar(255) DEFAULT NULL,
  `BYTEARRAY_ID_` varchar(64) DEFAULT NULL,
  `DOUBLE_` double DEFAULT NULL,
  `LONG_` bigint DEFAULT NULL,
  `TEXT_` varchar(4000) DEFAULT NULL,
  `TEXT2_` varchar(4000) DEFAULT NULL,
  `META_INFO_` varchar(4000) DEFAULT NULL,
  PRIMARY KEY (`ID_`) /*T![clustered_index] CLUSTERED */,
  KEY `ACT_IDX_RU_VAR_SCOPE_ID_TYPE` (`SCOPE_ID_`,`SCOPE_TYPE_`),
  KEY `ACT_IDX_RU_VAR_SUB_ID_TYPE` (`SUB_SCOPE_ID_`,`SCOPE_TYPE_`),
  KEY `ACT_IDX_VARIABLE_TASK_ID` (`TASK_ID_`),
  KEY `ACT_FK_VAR_BYTEARRAY` (`BYTEARRAY_ID_`),
  KEY `ACT_FK_VAR_BYTEARRAY` (`BYTEARRAY_ID_`),
  KEY `ACT_FK_VAR_EXE` (`EXECUTION_ID_`),
  KEY `ACT_FK_VAR_PROCINST` (`PROC_INST_ID_`),
  CONSTRAINT `ACT_FK_VAR_BYTEARRAY` FOREIGN KEY (`BYTEARRAY_ID_`) REFERENCES `ACT_GE_BYTEARRAY` (`ID_`),
  CONSTRAINT `ACT_FK_VAR_EXE` FOREIGN KEY (`EXECUTION_ID_`) REFERENCES `ACT_RU_EXECUTION` (`ID_`),
  CONSTRAINT `ACT_FK_VAR_PROCINST` FOREIGN KEY (`PROC_INST_ID_`) REFERENCES `ACT_RU_EXECUTION` (`ID_`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin

mysql> SELECT COUNT(*) FROM ACT_RU_VARIABLE WHERE TASK_ID_ IS NULL;
+----------+
| COUNT(*) |
+----------+
|  1747363 |
+----------+
1 row in set (0.56 sec)

mysql> SELECT COUNT(*) FROM ACT_RU_VARIABLE WHERE TASK_ID_ IS NOT NULL;
+----------+
| COUNT(*) |
+----------+
|        6 |
+----------+
1 row in set (0.61 sec)

2. What did you expect to see? (Required)

2026-01-28 13:45:48 (UTC+08:00)TiDB 10.170.15.90:4000[conn.go:1042] ["connection running loop panic"] [conn=4258500600] [session_alias=] [lastSQL="select * from ACT_RU_VARIABLE where task_id_='1997872469436538880'"] [err="runtime error: invalid memory address or nil pointer dereference"] [stack="github.com/pingcap/tidb/pkg/server.(*clientConn).Run.func1 /workspace/source/tidb/pkg/server/conn.go:1045\nruntime.gopanic /usr/local/go/src/runtime/panic.go:791\ngithub.com/pingcap/tidb/pkg/executor.(*Compiler).Compile.func1 /workspace/source/tidb/pkg/executor/compiler.go:59\nruntime.gopanic /usr/local/go/src/runtime/panic.go:791\nruntime.panicmem /usr/local/go/src/runtime/panic.go:262\nruntime.sigpanic /usr/local/go/src/runtime/signal_unix.go:917\ngithub.com/pingcap/tidb/pkg/planner/cardinality.estimateRowCountWithUniformDistribution /workspace/source/tidb/pkg/planner/cardinality/row_count_index.go:422\ngithub.com/pingcap/tidb/pkg/planner/cardinality.equalRowCountOnIndex /workspace/source/tidb/pkg/planner/cardinality/row_count_index.go:512\ngithub.com/pingcap/tidb/pkg/planner/cardinality.getIndexRowCountForStatsV2 /workspace/source/tidb/pkg/planner/cardinality/row_count_index.go:267\ngithub.com/pingcap/tidb/pkg/planner/cardinality.GetRowCountByIndexRanges /workspace/source/tidb/pkg/planner/cardinality/row_count_index.go:85\ngithub.com/pingcap/tidb/pkg/planner/core.detachCondAndBuildRangeForPath /workspace/source/tidb/pkg/planner/core/stats.go:411\ngithub.com/pingcap/tidb/pkg/planner/core.fillIndexPath /workspace/source/tidb/pkg/planner/core/stats.go:202\ngithub.com/pingcap/tidb/pkg/planner/core.deriveStats4DataSource /workspace/source/tidb/pkg/planner/core/stats.go:142\ngithub.com/pingcap/tidb/pkg/planner/core/operator/logicalop.(*DataSource).DeriveStats /workspace/source/tidb/pkg/planner/core/operator/logicalop/logical_datasource.go:364\ngithub.com/pingcap/tidb/pkg/planner/core/operator/logicalop.(*BaseLogicalPlan).RecursiveDeriveStats /workspace/source/tidb/pkg/planner/core/operator/logicalop/base_logical_plan.go:211\ngithub.com/pingcap/tidb/pkg/planner/core/operator/logicalop.(*BaseLogicalPlan).RecursiveDeriveStats /workspace/source/tidb/pkg/planner/core/operator/logicalop/base_logical_plan.go:204\ngithub.com/pingcap/tidb/pkg/planner/core.physicalOptimize /workspace/source/tidb/pkg/planner/core/optimizer.go:1027\ngithub.com/pingcap/tidb/pkg/planner/core.doOptimize /workspace/source/tidb/pkg/planner/core/optimizer.go:271\ngithub.com/pingcap/tidb/pkg/planner/core.DoOptimize /workspace/source/tidb/pkg/planner/core/optimizer.go:321\ngithub.com/pingcap/tidb/pkg/planner.optimize /workspace/source/tidb/pkg/planner/optimize.go:530\ngithub.com/pingcap/tidb/pkg/planner.Optimize /workspace/source/tidb/pkg/planner/optimize.go:361\ngithub.com/pingcap/tidb/pkg/executor.(*Compiler).Compile /workspace/source/tidb/pkg/executor/compiler.go:102\ngithub.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt /workspace/source/tidb/pkg/session/session.go:2112\ngithub.com/pingcap/tidb/pkg/server.(*TiDBContext).ExecuteStmt /workspace/source/tidb/pkg/server/driver_tidb.go:291\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).handleStmt /workspace/source/tidb/pkg/server/conn.go:2033\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).handleQuery /workspace/source/tidb/pkg/server/conn.go:1780\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).dispatch /workspace/source/tidb/pkg/server/conn.go:1379\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).Run /workspace/source/tidb/pkg/server/conn.go:1148\ngithub.com/pingcap/tidb/pkg/server.(*Server).onConn /workspace/source/tidb/pkg/server/server.go:741"]

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

tidb version is V8.5.4

Activity

  1. caiyifeng-1994
    added
    type/bugThe issue is confirmed as a bug.
    on Jan 28, 2026
  2. ti-chi-bot
    added
    contributionThis PR is from a community contributor.
    first-time-contributorIndicates that the PR was contributed by an external member and is a first-time contributor.
    on Jan 28, 2026

  3. ti-chi-bot commented on Jan 28, 2026

    @ti-chi-bot
    ti-chi-botBot
    on Jan 28, 2026 – with ti-chi-bot

    Welcome @caiyifeng-1994! It looks like this is your first issue to pingcap/tidb 🎉

  4. jebter
    added
    sig/plannerSIG: Planner
    severity/major
    impact/panic
    on Jan 29, 2026
  5. ti-chi-bot
    added
    may-affects-6.5
    may-affects-7.1
    may-affects-7.5
    may-affects-8.1
    may-affects-8.5
    on Jan 29, 2026
  6. qw4990
    removed
    severity/major
    may-affects-6.5
    may-affects-7.1
    on Jan 30, 2026

  7. 8 remaining items

  8. qw4990
    added
    affects-8.5This bug affects the 8.5.x(LTS) versions.
    on Feb 13, 2026
  9. hawkingrei
    self-assigned this
    on Feb 13, 2026
  10. ti-chi-bot
    added a commit that references this issue on Feb 14, 2026

    planner: fix a nil pointer bug caused by nil TopN (#66251)

    f23a3e7

  11. qw4990 commented on Feb 25, 2026

    @qw4990
    qw4990
    on Feb 25, 2026
    Contributor

    Fixed by #66251

  12. qw4990
    closed this as completedon Feb 25, 2026

  13. caiyifeng-1994 commented on Mar 12, 2026

    @caiyifeng-1994
    caiyifeng-1994
    on Mar 12, 2026
    Author

    已通过#66251修复

    It seems that this issue is caused by the database index "ACT_IDX_VARIABLE_TASK_ID". Is there any temporary solution to avoid this error?

  14. ti-chi-bot
    added a commit that references this issue on Mar 18, 2026

    planner: fix a nil pointer bug caused by nil TopN (#67106)

    be2c956

  15. qw4990 commented on Apr 16, 2026

    @qw4990
    qw4990
    on Apr 16, 2026
    Contributor

    Reproduce this:

    SET SESSION tidb_stats_load_sync_wait = 0;
DROP TABLE IF EXISTS t_spec;
CREATE TABLE t_spec (
id bigint NOT NULL AUTO_INCREMENT,
product_id bigint NOT NULL,
shop_id varchar(10) NOT NULL,
topic_id char(36) DEFAULT NULL,
spec_name varchar(30) NOT NULL,
spec_value text DEFAULT NULL,
buy_flag char(1) DEFAULT NULL,
relation char(1) NOT NULL,
creator_id varchar(50) DEFAULT NULL,
creator_name varchar(50) DEFAULT NULL,
create_time datetime DEFAULT CURRENT_TIMESTAMP,
update_time datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
package_id bigint DEFAULT NULL,
PRIMARY KEY (id),
KEY index_product (product_id),
KEY index_package (package_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;


INSERT INTO t_spec
(product_id, shop_id, spec_name, spec_value, buy_flag, relation, creator_id, creator_name, package_id)
VALUES
(1, 'XXX', 'a', 'x', '1', '1', 'u1', 'u1', NULL),
(2, 'XXX', 'b', 'y', '1', '1', 'u1', 'u1', NULL),
(3, 'XXX', 'c', 'z', '1', '1', 'u1', 'u1', 123);


ANALYZE TABLE t_spec;
SHOW STATS_HISTOGRAMS WHERE table_name='t_spec';
SHOW STATS_TOPN WHERE table_name='t_spec';

EXPLAIN SELECT id, product_id, shop_id, topic_id, spec_name, spec_value, buy_flag, relation,
creator_id, creator_name, create_time, update_time, package_id
FROM t_spec
WHERE package_id = 123 AND shop_id = 'XXX'
ORDER BY spec_name ASC;
  16. qw4990
    added a commit that references this issue on Apr 17, 2026

    planner: fix a nil pointer bug caused by nil TopN (pingcap#67106)

    7839ed1
  17. ti-chi-bot
    added a commit that references this issue on Apr 17, 2026

    planner: fix a nil pointer bug caused by nil TopN (#67840)

    d7709be

  18. terry1purcell commented on Jun 19, 2026

    @terry1purcell
    terry1purcell
    on Jun 19, 2026
    Contributor

    Using @qw4990's instructions above may not be enough. Here are the additional steps to reproduce and fix the issue:

    I added another index:

    create index index_topic on t_spec (topic_id);

    Then I repeated the following SQL until I hit 700,000 rows:

    INSERT INTO t_spec (product_id, shop_id, spec_name, spec_value, buy_flag, relation, creator_id, creator_name, package_id) select product_id, shop_id, spec_name, spec_value, buy_flag, relation, creator_id, creator_name, package_id from t_spec;

    Then I inserted a final dummy row where topic_id is not null:

    tidb> INSERT INTO t_spec (product_id, shop_id, spec_name, spec_value, buy_flag, relation, creator_id, creator_name, package_id, topic_id) select product_id, shop_id, spec_name, spec_value, buy_flag, relation, creator_id, creator_name, package_id, 'XXX' from t_spec where id = 1;

    Next:

    ANALYZE TABLE t_spec:

    I then ran this update (which is intended to target user tables with modifyCount == 0 and sampling (which kicks in at 100,000 rows):

    update mysql.stats_meta set modify_count = 1 where table_id > 100 and count > 100000 and modify_count = 0;

    Then you should be able to reproduce the panic using this version of the SQL:

    EXPLAIN SELECT id, product_id, shop_id, topic_id, spec_name, spec_value, buy_flag, relation, creator_id, creator_name, create_time, update_time, package_id FROM t_spec WHERE topic_id = 'XXX' ORDER BY spec_name ASC;

    To reload stats_meta and fix the panic.

     SET @v = (SELECT MAX(version) + 1 FROM mysql.stats_meta);
update mysql.stats_meta set modify_count = 1, version=@v where table_id > 100 and count > 100000 and modify_count = 0;

  19. qw4990 commented on Jun 20, 2026

    @qw4990
    qw4990
    on Jun 20, 2026
    Contributor

    Setting modify_cnt to 1 should be able to keep the optimizer out from the panic logic:

    Image

  20. qw4990 commented on Jun 20, 2026

    @qw4990
    qw4990
    on Jun 20, 2026
    Contributor

    Another case that can reproduce this panic:

    DROP DATABASE IF EXISTS repro65867_act;
CREATE DATABASE repro65867_act;
USE repro65867_act;

SET SESSION tidb_stats_load_sync_wait = 0;

CREATE TABLE ACT_RU_VARIABLE (
  ID_ varchar(64) NOT NULL,
  REV_ int DEFAULT NULL,
  TYPE_ varchar(255) NOT NULL,
  NAME_ varchar(255) NOT NULL,
  TASK_ID_ varchar(64) DEFAULT NULL,
  TEXT_ varchar(4000) DEFAULT NULL,
  PRIMARY KEY (ID_) CLUSTERED,
  KEY ACT_IDX_VARIABLE_TASK_ID (TASK_ID_)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;

INSERT INTO ACT_RU_VARIABLE(ID_, REV_, TYPE_, NAME_, TASK_ID_, TEXT_) VALUES
  ('1', 1, 'string', 'a', NULL, 'x'),
  ('2', 1, 'string', 'b', NULL, 'y'),
  ('3', 1, 'string', 'c', '1997872469436538880', 'z');

ANALYZE TABLE ACT_RU_VARIABLE;

SELECT * FROM ACT_RU_VARIABLE
WHERE task_id_ = '1997872469436538880';
  21. ti-chi-bot
    added a commit that references this issue on Jul 1, 2026

    planner: fix a nil pointer bug caused by nil TopN (#66251) (#69570)

    d371098
  22. ti-chi-bot
    added a commit that references this issue on Jul 10, 2026

    planner: fix a nil pointer bug caused by nil TopN (#66251) (#69750)

    7800a59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

affects-8.5This bug affects the 8.5.x(LTS) versions.contributionThis PR is from a community contributor.first-time-contributorIndicates that the PR was contributed by an external member and is a first-time contributor.impact/panicreport/customerCustomers have encountered this bug.severity/moderatesig/plannerSIG: Plannertype/bugThe issue is confirmed as a bug.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions