@spyced writes about evaluating Jev, Gemini, and OpenAI decision models, and why they picked GLiDE, from a much smaller provider, for Mjolnir's GitHub event classifications.
- 468 views
- 6 likes
- 2 bookmarks
- 1 reposts
About this case
Benchmark. Jonathan Ellis (@spyced) links https://blog.slopcop.com/comparing-jev-gemini-and-openai-decision-models/ . The post says Mjolnir delivers relevant GitHub events so agents stay looped in with the rest of the team. Activity on your own issues or pull requests is always relevant. Events they cannot classify programmatically go to a classifier. They started with Jev because it was there first, then ran a bakeoff after Google released Gemini Decisions, and chose GLiDE from Fastino Labs: the best results for their use case, and one of the cheapest options. The only real ding he names is speed: GLiDE was the second-slowest service tested, and Gemini and OpenAI were where speed was strongest. He also writes that a first label set, in which a strong adjudicator saw a pair only when the first labeller disagreed with Jev, made Jev's recall look like 0.91, and that against an independent panel it is 0.57. If there is no time or data for a proper eval, he says Jev is a good place to start: inexpensive, smart, and reasonably fast. He calls the weights open-ish, and says commercial use needs a separate agreement if annual revenue is over $10M. Text only.
Original posthttps://x.com/spyced/status/2120954962926834145