Case wall
Jev AI Benchmarks cases
31 public posts filed under Benchmarks. English summaries on this page, with a link to each case.
Last updated
These are the public posts filed under the Benchmarks category.
No posts match these filters.
@yonyoniz tests Jev on Hebrew tweets: same stance as English on 8 of 9; language isn't the bottleneck.
Does Jev read Hebrew? I tested it on my own tweets https://x.com/i/article/2113860659519598592 Liran Tal asked a fair question: does TypeSafe's small decision m…
@0xasten classified 64,946 GitHub repos with TypeSafe Jev for under $2, ~90% accurate on 170 hand labels.
I classified 64,946 GitHub repos using @typesafeai's Jev — for under $2. Here is what I found: 🤖 Agents are dominating the AI space Agents alone (3,029) outnumb…
@honnibal ran Jev with Ellf on data from Opus few-shot classifiers; Jev agreed with Opus on 95% of the inputs.
Very impressed with @typesafeai Jev. We've been working on a user project that was built with a lot of few-shot classifiers powered by Opus. I used Ellf to run…
@ikkistan benchmarks guest-photo moderation for Candids on UnsafeBench: Cloudflare clef-flash, TypeSafe Jev, Claude Haiku 5.5, and Amazon Rekognition.
I’m building Candids, an app for wedding and party guests add photos to one shared album from their phones. Guest uploads need moderation, so I benchmarked four…
@ikkistan: Jev, Haiku, and clef-flash reach AUROC 0.86 to 0.87 on unsafe images, Rekognition 0.70, at 4 to 18 cents per 1,000 images versus $1.
Jev, Haiku and clef-flash rank unsafe images about equally well (AUROC 0.86 to 0.87, Rekognition 0.70) at 4 to 18 cents per 1,000 images vs $1. Code, per-image…
@instantricecook runs 66 Where's Waldo scenes across decision models; Jev 1.14 is quickest, with a public Waldo bench site.
I wanted to see how well decision models are at... Finding Waldo. After 66 scenes, here are the results: - @liquidai d1-ultra found the most Waldos - @typesafea…
@gcai: GC AI Engineering built 38 projects in 12 hours with TypeSafe Jev while @ChrisDzoba tried to break it.
GC AI Engineering built 38 projects in 12 hours with TypeSafe's Jev, a classification model. Director of Engineering @ChrisDzoba spent the day trying to break i…
@SaadMRP: Jev diagnoses SREGym-Lite faults; LLM review lifts pass rate 78.1%→89.5% across 21 faults, ~38s avg.
1/8 In our latest experiment, we let Jev, a decision model, investigate system failures and asked an LLM to check its diagnosis. Across 21 SREGym-Lite faults, r…
@gbolagadehq compares Jev 1.14, OpenAI Decisions API, and Opus 5.5 on five public tests including 1,999 product photos.
https://x.com/i/article/2112290824528576512
@a_latner benchmarks three-day-old Jev image classification against Haiku 5.5 on real production data and reports matching quality.
@typesafeai Jev image classification is 3 days old. I benchmarked it against Haiku 5.5 on real production data. We know it's faster, but are the results as good…
@huuw_khoa benchmarks Jev against Gemma 4 12B and gpt-oss-20b on 60 knowledge-graph questions in open-source hopwise.
Can a model built for typed decisions walk a knowledge graph better than a general LLM? I tried Jev, Gemma 4 12B and gpt-oss-20b on 60 questions. Feedback and i…
@Adityakuma8784 built jev-harness to test coding-agent guardrails; a typical hook let 331 of 503 disguised dangerous commands through.
Your AI coding agent has guardrails: hooks, permission rules, CLAUDE.md, AI reviewers. Almost nobody tests them. I built jev-harness to test them. Against a typ…
@_vMyth’s jevals runs agent evals and guardrails as one Jev decision request per trace, cheap enough to score every run.
Runs agent evals and guardrails as 1 request to a Jev decision model, not an LLM judge. Cheap enough to score every trace. Change 2 defaults: the quick-start se…
@OpenRouter posted Decision Model Rankings for spend share and token share by model and task type, and says @typesafeai is leading all categories today.
New: Decision Model Rankings! See the spend share and token share of different decision models and task types, including Relevance, Correctness, Instruction Fol…
@h_nilforoshan benchmarked the OpenAI Decisions API against Jev for HiringCafe.com: relevance of a query or resume to a job, scored 1-10. The post says OpenAI costs 2x more and is 5-10% worse.
I benchmarked "Jev-killer" OpenAI Decisions API against Jev for https://HiringCafe.com, serving 2.5 million users. The task is to predict how relevant a user qu…
Quoted @OpenAIDevs
@ValsAI says they independently tested TypeSafe's Jev against 11 other models. On 400 claim-verification questions it matched GPT-6 Astra's 97.5% accuracy at about 1/500th the cost.
Can a model that doesn’t generate text compete with frontier LLMs? We independently tested TypeSafe’s Jev against 11 other models. On 400 claim-verification que…
@rox_ai swapped LLM reranking for Jev classification on sales context retrieval and reports 20× faster, 10× cheaper, and 12% more accurate than GPT-5 Mini.
We used Jev to retrieve sales data 20x faster, 10x cheaper, and 12% more accurate than GPT-5 Mini. Every time a Rox agent answers a query, it pulls from relevan…
@LangChain says they tested Jev against LLM judges on accuracy, repeatability, latency, and cost, to see whether System One models could be a new approach to agent evaluation. They link a write-up.
We tested Jev against LLM judges on accuracy, repeatability, latency, and cost to see whether System One models could offer a new approach to agent evaluation.…
@airesearch12 says JevBench results are in, and that Jev is still in the lead but the race is close. The post does not list the other scores.
JevBench results are in. Jev still in the lead, but it's close.
@Layton_Gott reports that Cua built a 2.8MB, 706K-parameter model scoring 99.7% on their form-filling eval, while hosted Jev scored 83.6%. They say the small model is free, runs locally in under 1GB of RAM, and decides in one pass.
Yesterday I said Jev would open a ton of doors... 24 hours later, this exists. Cua built a 2.8MB model that scored 99.7% on their form-filling eval. Hosted Jev…
Quoted @trycua
@OpenMed_AI ran four typed questions on four authored fictional notes: Jev 16/16 vs Laya 13/16, calling it a useful diagnostic not a clinical claim.
Four typed questions across four authored fictional notes, labels written before the calls. Jev 16/16, Laya 13/16. On the medication-conflict question alone: 4/…
@ego_agent compares ego lite plus Jev plus DeepSeek Flash with GPT-5.6 Sol on 20 Amazon product decisions. They report 3.71s versus 54.45s, and 10/10 on both.
ego lite + Jev + DeepSeek Flash = stupid fast. ⚡ 3.71s for 20 Amazon product decisions. GPT-5.6 Sol: 54.45s Same task. Same result. 10/10 on both. Less reasonin…
Quoted @CompleteSkeptic
@nutlope classified 100 emails (50 legit, 50 fraudulent) with Jev in 1.42 seconds, then sent the 31 predictions under 95% confidence to Kimi K3. The combined pipeline was 96/100 correct in 16 seconds for about $0.07, of which about $0.003 was Jev.
Jev + Kimi K3 for fraud detection! TLDR: Jev classified 100 emails in 1.42 seconds, then I routed the uncertain cases to Kimi K3. The full pipeline got 96/100 c…
@aimlapi had Jev V13 play 5+0 blitz chess, one API call per move, against Fable 5.1 and GPT-6 Astra. They report Fable was up a queen and +16 material by move 29 but lost on time because it spent 6–15 seconds per move while Jev answered in about 2.6 seconds; Astra checkmated Jev in 18 moves (Qe1#) with 2:27 left.
Jev VS Fable 5.1 VS GPT-6 Astra: chess typesafe's new Jev V13 isn't an LLM. it doesn't chat, doesn't explain, doesn't write code — it only makes decisions. so w…
Quoted @CompleteSkeptic