Pinned
Before you write good evals, you have to pick the right things to evaluate. Main take-aways:
- source metrics from failures you see in traces
- keep the set small and maintainable
- only keep a metric if you'd take action when it moves
Article
How to build an eval set you can maintain
"I have traces, how do I set up evals?" is a very common question, and an important one to get right. This article will guide you through choosing the right metrics to evaluate.
Three kinds of metrics
