Testing a cheap model on my own agent's decisions
I tested a small decision model on 555 real choices from my own agent. It lost.
I followed a public recipe for using your own logs to try out a new workflow. I pulled 555 routing decisions from my own AI agent and asked the Jev decisions model to make the same calls.
Jev agreed with my agent 24.5 percent of the time. Always giving the most common answer scored 38.2 percent, so Jev and every other model I tried lost to that simple rule.
Jev was the cheapest and the fastest. When I kept only its most confident answers, it agreed 69.4 percent of the time, on about 15 percent of the cases.
I wrote a checker that recomputes every number in the report from the raw data.
- Cases555 routing decisions from my own agent
- Agreement24.5 percent (simple rule 38.2)
