[yoanbernabeu]

Summary

Adds embedder.max_batch_size and embedder.max_batch_tokens (openai provider). OpenAI-compatible endpoints (vLLM, TEI, proxies, Ollama /v1…) often time out or fail on the OpenAI-sized batches (2000 inputs / 280000 estimated tokens); users had to put a splitting proxy in front.

embedder:
  provider: openai
  endpoint: https://embeddings.example.com/v1
  max_batch_size: 100
  max_batch_tokens: 50000

Changes

• config: two omitempty fields; unset/0 keeps the previous limits.
• embedder/batch.go: BatchLimits, DefaultBatchLimits(), FormBatchesWithLimits() (FormBatches unchanged, uses the defaults), optional BatchLimiter interface.
• OpenAIEmbedder: WithOpenAIBatchLimits option, BatchLimits(), and EmbedBatch (watcher path, one file per call) now splits into several requests when a file exceeds the limits, preserving order.
• indexer: IndexAll forms batches with the embedder's limits.
• Docs (configuration.md, backends/embedders.md) and CHANGELOG.
• Tests: custom limits (size, tokens, oversized single chunk sent alone), defaults identical to FormBatches, EmbedBatch splitting + ordering, factory wiring, YAML load, and default config not writing the new keys.

Test Plan

• make lint: 0 issues. go test -race ./...: all green.
• Real project (V1 rule): grepai itself (Go, 348 files, 1706 chunks), provider: openai pointed at Ollama's OpenAI-compatible endpoint (nomic-embed-text) through a request-logging proxy:

	main (defaults)	this PR (max_batch_size: 50, max_batch_tokens: 20000)
Largest request	607 inputs, ~279k est. tokens	48 inputs, 19,975 est. tokens
Result	❌ indexing fails (endpoint resets the connection)	✅ 41 requests, 1706 chunks indexed

Watcher: adding a file with 84 new chunks under grepai watch produced 3 requests (38/38/8 inputs, ≤ 20000 est. tokens).

• Backward compatibility: with a config without the new keys, against a stub endpoint, main and this PR send byte-identical batches (495 / 604 / 607 inputs). Default config does not gain the new keys on save (tested).

Closes #92

🤖 Generated with Claude Code

https://claude.ai/code/session_01GNtGZcXfz7hyZsrgHrdzuj