@amzslaw moved a multi-label classifier from an LLM judge onto TypeSafe Jev; after question and cutoff tuning it matched LLM accuracy and ran faster.
Amy Slawson @amzslaw
EN
- 101 views
- 0 likes
- 0 bookmarks
- 0 reposts
Is the cat plotting revenge? 😼 We moved a classifier off an LLM judge and onto Jev, TypeSafe's decision model. It matched the LLM on accuracy and runs much fast…
About this case
Guide with X article and charts. Amy Slawson retells switching a pet-psychic-style classifier from SetFit to an LLM judge to Jev. Lessons: build a real eval, tune question wording then per-label cutoffs, cross-validate cutoffs, and look past headline F1 (precision depends on base rate). Jev matched or beat the LLM on most labels, about 20% cheaper, with much higher concurrency.
Original posthttps://x.com/amzslaw/status/2115858230849257898