Jev as a judge: rubric scoring and evaluation

What this page is

This page does not add a new benchmark. It points at posts already in the case catalog. Score on this site is an ordered scale (Ordered scale: at least 2 and at most 10 levels) and returns { type, score, legend, probabilities, confidence }. Rubric judging, on the Jev vs LLM page, means a score or choice over fixed criteria when you want a distribution instead of a paragraph.

Read Jev AI vs LLM and the primitives field reference. Recorded model ids are on Jev AI on OpenRouter.

OpenRouter traffic stored here

OpenRouter traffic names stored in this handbook include sel-jev-rubric-judge (9.18B tokens). That label is not a GitHub URL in the project directory.

Posts that mention a judge, a rubric, or an eval

Each line is the English summary already stored for that post. The language button can load the Japanese, Korean, or Chinese summary for these lines. 28 posts currently match.

• @NFT_Chen@NFT_Chen announces DeepEval JevEval: Jev as a custom eval judge for Yes/No, Score, Choice, and agent traces.
• @HuanxinShe5254@HuanxinShe5254 ships JevAny-Robotics-4B, a robotics Jev that judges progress, success, and failure in 185ms.
• @tonysimons@tonysimons ships Verdict: a second AI on Grok Bot that uses TypeSafe Jev to rank X posts and drafts against your criteria.
• @parkerbrowndev@parkerbrowndev hooks Jev before tool calls in a Pi/Harbor setup with Haiku; reports better safety, half the tokens, higher API $ from broken caching.
• @sinkush@sinkush: a local hybrid pass takes the top 30, then Jev judges each hit in parallel in about 0.3 s for a tenth of a cent.
• @zhining_liu@zhining_liu launches JevAny: OSS toolkit to train/deploy System-1 decision models; 27B release beats Jev 1.13.0 on two evals.
• @_vMyth@_vMyth’s jevals runs agent evals and guardrails as one Jev decision request per trace, cheap enough to score every run.
• @Asteri_eth@Asteri_eth listed ten GitHub repos already shipping on Jev, from a browser agent to a Claude Code bridge and a LangGraph router.
• @_avichawla@_avichawla explains standard RAG (chunk → embed → retrieve → optional rerank → LLM) versus adding Jev into the RAG path, with an animated explainer video.
• @Sumanth_077@Sumanth_077 built a data-analyst agent where Qwen writes SQL/answers and Jev reviews question clarity, SQL fit, and grounded explanations; open-sourced under Hands-On-AI-Engineering.
• @Sumanth_077@Sumanth_077 explains jevgrep: Jev judges which folders, files, and declarations are relevant, then returns source excerpts for the coding agent.
• @rasukarusan2@rasukarusan2 says they can now use Jev to judge whether someone is a hype account (驚き屋).
• @miyagawa@miyagawa gives Jev a transcript and show notes so it can judge at what minute and second each link topic appears. They say a 3-hour episode took 0.5s and $0.003.
• @akshay_pachaar@akshay_pachaar links an article titled “Build a Jev Judge.”
• @LangChain@LangChain says they tested Jev against LLM judges on accuracy, repeatability, latency, and cost, to see whether System One models could be a new approach to agent evaluation. They link a write-up.
• @Layton_Gott@Layton_Gott reports that Cua built a 2.8MB, 706K-parameter model scoring 99.7% on their form-filling eval, while hosted Jev scored 83.6%. They say the small model is free, runs locally in under 1GB of RAM, and decides in one pass.
• @robj3d3@robj3d3 describes a doomscrolling filter: you pick a niche, Jev reads 3 days of posts and asks 8 questions each, in about 2 seconds for $0.007. It removes bait and hidden ads and judges text plus like, reply, and repost ratios. They say it is free with no signup.
• @kunchenguid@kunchenguid introduces compact-adviser, an agent plugin for Claude and Pi that uses Jev to judge whether a session is at a task boundary safe to compact. They built a private eval from 40 real sessions and say the classifier favors precision when the window is small and recall as it fills up.
• @akira_papa_IT@akira_papa_IT built a chat app by making Jev judge the next character. They say it is very fast, input is $0.042 per 1M tokens, and output is free without a cap.
• @alin_zone@alin_zone says Jev fits Codex computer use: use Jev as the judge that tells Codex how to operate the computer quickly while saving tokens. They recorded a video of how to apply for access on the official site.
• @waynesutton@waynesutton launched askjev.ai: you can ask Jev anything and it will judge rather than answer. They invite people to try for 1 million questions and say TypeSafe and Convex work together.
• @edteamlat@edteamlat explains Jev as a new AI from Diogo Almeida, a ChatGPT co-creator, claiming up to 200x faster and 400x cheaper than current models. Unlike an LLM, it does not generate text word by word: it evaluates options, decides, and returns probabilities.
• @nidhisinghattri@nidhisinghattri built a model-router CLI with Jev: you give a task and the AI subscriptions you have, and Jev says which agent or model should take it. They say pairing it with Herdr is the point.
• @vladdubchak_x@vladdubchak_x built a Maxfusion plus TypeSafe tool that scores every shot of a Meta video ad in 1.5 seconds. They pushed 450-plus ads through in under 3 minutes, and say one Duolingo ad received 50 judgments in 1.52 seconds for $0.00028 and a score of 88.
• @yelkhayami@yelkhayami built an extension that judges tweets as they are typed. They say it reacts to each keystroke and that an hour of testing cost half a cent.
• @kzkhykw@kzkhykw used TypeSafe Jev to judge local Whisper transcripts and automate a Mac. Chrome pages use a browser harness; Slack and Notion are seen and operated through Mac accessibility. They say the implementation was done with Grok 4.6 only.
• @Mnilax@Mnilax wired Jev into the routing layer of a build that had been running on Astra and says the same task finished 37 minutes sooner with no malformed responses to retry. The post also cites TypeSafe’s published figures: $0.042 per million input tokens with free output, 67.8% matching Sonnet 5 on their eval at 294x less cost and 195x the speed, 0.114s versus 8.566s for GPT-5.6 Terra on the same demo, and about 10 decisions per second in the Doom demo.
• @parcadei@parcadei says that after a failed attempt, use Jev to judge whether the retry is meaningfully different; otherwise replan, escalate, or stop.

The full catalog is Jev AI cases. GitHub repositories are on Jev GitHub repositories.