I built a repeatable way to check whether token-saving tools actually work

Tue, 28 Jul 2026 17:24:03 +0000

Hi, I’m Yu. Earlier this month I released Tura and wrote about why agent harnesses need long-horizon benchmarks, not just one-off demos.

Over the past two weeks, I’ve seen more people publish tests of token-saving tools such as RTK and Ponytail. That’s great, but the results are hard to compare when everyone uses different tasks, logs, and output formats.

So I open-sourced the benchmark framework I use:

https://github.com/Tura-AI/benchmark/tree/main

It can run the benchmark workflow, collect logs and artifacts into one result schema, and let CI index any result added to the repository. The website then turns those results into comparable graphs and detailed run pages.

If you have a tool, harness, or test case you want included, tell me. You can also run the benchmark locally, reproduce the evaluation, and submit the result folder as a pull request. CI handles the indexing automatically.

I’d especially like feedback on what should be tested next and where the framework may introduce bias.

Disclosure: I maintain Tura and this benchmark framework.

Coder Legion - Recent posts tagged agent-tools

I built a repeatable way to check whether token-saving tools actually work