[CI] Teach the CI system to automatically skip build/test jobs for diffs only modifying code comments #8691

Overview

I'd like to consider an automated way to skip CI if a diff only modifies code comments within source files.

What we have today

We have code that automatically skips CI if only certain file patterns are in the diff, e.g. only markdown files, CODEOWNERS, etc.:

CI can also be skipped by applying the ci:skip label manually on pull requests, though the label does not apply to push events of those diffs.

Potential benefits / scoping

From an agent:

This occurs often enough to justify a small checker, but the opportunity looks modest: about 0.5% of sampled commits. I’d start with conservative comment-edit detection and measure its decisions before enabling automatic skipping.

History results

I examined first-parent history reachable from the local checkouts for August 1–September 28, 2026:

Repository	Commits examined	Already covered by current path filters	Additional comment-only candidates
rocm-systems	1,614	134	10 — 0.62%
rocm-libraries	1,360	180	5 — 0.37%
Total	2,974	314	15 — 0.50%

“Candidates” means all changes outside currently skippable paths were source comments. Some also changed documentation files. Two candidates were copyright/license edits; the other 13 were explanatory comments or API documentation.

Examples:

Repository / change	Source changes
ROCm/rocm-systems#11670	Add a hipHostRegister documentation warning
ROCm/rocm-systems#11437	Fix GPU-isolation URLs in header and implementation comments
ROCm/rocm-systems#9481	Remove a stale Python comment
ROCm/rocm-libraries#11103	Correct Cholesky formulas in API comments
ROCm/rocm-libraries#10681	Replace stale TODOs with explanatory comments
ROCm/rocm-libraries#10486	Correct supported data types in a header comment

Implementation ideas

Separate GitHub App/bot that applies labels to pull requests

I add the ci:skip label myself when I author or review changes that I judge to have no impact on CI, we could have an AI agent do this too. Some risk of false positives, but less code to maintain and could handle particularly tricky edge cases that are hard to directly code for.

Inspect the diff as part of CI configuration at job start time

This will require having the full diff, not just files modified, available. We don't currently have that information in component repositories but it should be possible to fetch within a reasonable budget (< 10 seconds).

Ideas from an agent:

Approach	Assessment
Regex over added/deleted lines	Useful prefilter; unsafe as the final decision
Small C-family lexer + Python tokenize	Recommended starting point
Tree-sitter comment spans	Reasonable alternative, with grammar dependencies and parse-error handling
Compiler preprocessing / AST comparison	Too much configuration and toolchain complexity for this setup-stage optimization

The distinction I’d enforce is “approved comment edits for these build/test workflows,” rather than “comments cannot affect CI.”

Concrete reasons:

• // RUN:, CHECK:, and XFAIL: can control tests. The local hipThreads tree contains these directives; LLVM documents their interpretation. LLVM testing guide
• C/C++ backslash-newline processing precedes comment removal. Raw strings can also contain apparent comment syntax. GCC preprocessing
• Python encoding declarations are comments with language semantics. Docstrings are strings and should remain significant. Python lexical rules
• Added comment lines can change __LINE__, source locations, diagnostics, and traceback information.

The warning about // RUN lines in particular is worth heeding. We'd need to be careful with allow-lists and deny-lists for patterns/files/paths/subprojects.

Appendix - A general solution

Build tools that look at compilation units and AST graphs of source files in compiled languages can determine quickly if a change to a source file needs a rebuild, but this is harder for CI systems that aren't integrated into remote build environments (e.g. Bazel with remote build execution).