[kirillk]

Issue

No existing issue. This was reported directly by a user of the IntelliJ plugin who hits "Invalid ripgrep match output" on projects that contain ISO-8859-1 / Latin-1 encoded files.

What Problem This Solves

When a repository contains files that are not valid UTF-8, such as ISO-8859-1 / Windows-1252 source, Kilo's grep frequently failed with Invalid ripgrep match output. Any search that matched one of those lines failed entirely, and the agent had to fall back to slower shell-based searches. This affected the CLI, the VS Code extension and the IntelliJ plugin, since they all run ripgrep through the same core service.

Why This Change Was Made

Ripgrep reports non-UTF-8 data differently in its --json output. Instead of {"text": "..."} it emits {"bytes": "<base64>"}. A single Latin-1 byte (for example 0xF6 for ö) is enough to trigger this. The grep parser only accepted the text shape, so Schema.decodeUnknownEffect(RawMatch) failed and the whole record was rejected.

Changes:

• The grep parser now accepts both the text and bytes shapes for the matched line, the file path, and each submatch, decoding the base64 bytes back to text.
• Base64 bytes are decoded as Windows-1252 (the WHATWG mapping for ISO-8859-1). Every byte maps to exactly one UTF-16 unit, so ripgrep's byte offsets still line up with the decoded text and message positions are unchanged for UTF-8 files, which never take this path.

Implementation

A new Data schema in packages/core/src/kilocode/ripgrep-grep.ts models ripgrep's "arbitrary data" type as either { text } or { bytes } and normalizes it to { text } via a decoding transform. packages/core/src/ripgrep.ts reuses it for data.lines, data.path, and submatches[].match. The change to the shared upstream file is limited to those three field types, each marked with kilocode_change.

Read and edit still do not decode non-UTF-8 files (read rejects them with MalformedUtf8Error; edit decodes lossily), so the agent can still fall back to shell for those operations. Fixing that needs encoding detection and round-tripping, and is deliberately out of scope for this PR.

User Impact

Grep now works normally on projects that mix UTF-8 and Latin-1 / Windows-1252 files, and returns the correct characters instead of failing. UTF-8 files behave exactly as before.

Screenshots / Video

N/A - no visual changes.

How to Test

Manual/local verification

• Agent ran bun test ./test/kilocode/ripgrep-encoding.test.ts ./test/ripgrep.test.ts ./test/kilocode/grep-tool.test.ts ./test/kilocode/ripgrep-settlement.test.ts ./test/filesystem/search.test.ts in packages/core/: 19 pass, 0 fail. The four new encoding tests all failed with Invalid ripgrep match output before the fix.
• Agent ran bun run typecheck in packages/core/: passed.
• Agent ran bun run script/check-opencode-annotations.ts: passed.

Reviewer test steps

1. Create a file containing a raw Latin-1 byte, for example printf 'title = "Handl\xe4ggning"\n' > legacy.java.
2. Run a grep for Handl in that directory.
3. Confirm the match is returned with Handläggning in the text, instead of Invalid ripgrep match output.

Blocked checks and substitute verification

• None. All relevant checks ran.

Dependency

None.

Checklist

• Issue linked above, or exception explained
• Tests/verification described
• Screenshots/video included for visual changes, or marked N/A
• Changeset considered for user-facing changes
• I personally reviewed the diff and can explain the changes, including any AI-assisted work.

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

The incremental change (commit ba80d03) only adds a documentation caveat above Data explaining that submatch start/end are raw ripgrep byte offsets that no longer index into the decoded text on mixed-encoding lines. The caveat is accurate — nothing in-repo slices with these offsets today (the grep tool uses only text) — and it resolves the previously raised offset-alignment suggestion via the documented tradeoff, which the existing inline thread already covers.

Files Reviewed (1 file)

• packages/core/src/kilocode/ripgrep-grep.ts

Previous Review Summaries (3 snapshots, latest commit 31906a5)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 31906a5)

Status: 1 Issue Found | Recommendation: Address before merge

Overview

Severity	Count
CRITICAL	0
WARNING	0
SUGGESTION	1

Issue Details (click to expand)

SUGGESTION

File	Line	Issue
packages/core/src/kilocode/ripgrep-grep.ts	16	Lossy UTF-8 decode drops submatch offset alignment on mixed-encoding lines

Files Reviewed (2 files)

• packages/core/src/kilocode/ripgrep-grep.ts - 1 suggestion
• packages/core/test/kilocode/ripgrep-encoding.test.ts

Fix these issues in Kilo Cloud

Previous review (commit 7bf5378)

Status: No Issues Found | Recommendation: Merge

Files Reviewed (1 file)

• packages/core/src/kilocode/ripgrep-grep.ts

Incremental review of 0e6da310..7bf5378d: the only change since the last review is a documentation comment documenting the mixed-encoding decode tradeoff. It correctly describes the behavior (UTF-8 ä → ä when a line contains any invalid UTF-8 byte, while fully UTF-8 and fully Latin-1 lines are unaffected), so the previous SUGGESTION is resolved. No new or unresolved findings.

Previous review (commit 0e6da31)

Status: 1 Issues Found | Recommendation: Address before merge

Overview

Severity	Count
CRITICAL	0
WARNING	0
SUGGESTION	1

Issue Details (click to expand)

SUGGESTION

File	Line	Issue
packages/core/src/kilocode/ripgrep-grep.ts	16	ripgrep emits the whole lines value as base64 bytes when any byte in the line is invalid UTF-8, so valid UTF-8 multibyte sequences on the same line are decoded as Windows-1252 and become mojibake. Offset alignment is preserved, so this is a narrow tradeoff, but worth a note.

Files Reviewed (4 files)

• .changeset/grep-latin1-matches.md - no issues
• packages/core/src/kilocode/ripgrep-grep.ts - 1 suggestion
• packages/core/src/ripgrep.ts - no issues
• packages/core/test/kilocode/ripgrep-encoding.test.ts - no issues

The fix is sound: the new Data schema correctly accepts both ripgrep arbitrary data shapes and normalizes to { text }, the shared packages/core/src/ripgrep.ts change is limited to the three field types with kilocode_change markers, and the tests cover match lines, submatches, Windows-1252 punctuation, and context lines. No blocking issues.

Fix these issues in Kilo Cloud

---

_{Reviewed by deepseek-v4.1-flash · Input: 29.6K · Output: 3.5K · Cached: 210.9K}

_{Review guidance: REVIEW.md from base branch main}

[kirillk]

Tracked for the opencode v2 rewrite in #15087 (core-v1-changes-port).