@maharishi_r_b says they tested three AI decision models on a private eval: @SarvamAI Prism, @typesafeai Jev, and @Alphanimble Wity. At 5% false positives, Jev caught 77% of errors, Prism 59%, and Wity 57%. To catch 80% of issues, Jev needed 5.9% false positives, against 11.7% for Prism and 20% for Wity. In Sarvam’s launch benchmarks Prism scored 71.8 against Jev’s 64.5, and Alpha nimble said Wity was better than Sarvam, but that is not what this eval showed. The task has a significant mix of Indian languages, and Jev won.
- 181 views
- 2 likes
- 0 bookmarks
- 1 reposts
We tested 3 AI decision models on our private eval set: @SarvamAI - Prism @typesafeai - Jev & @Alphanimble - Wity Here’s what we found 🧵
About this case
Benchmark. Maharishi R B (@maharishi_r_b) opens a thread. On speed, Sarvam’s Prism was the clear winner: 0.37 s per decision against Jev’s 0.61 s, median, called from India. @alphanimble’s Wity was slower at 1.6 s, and he says that is not quite a fair comparison given their GPU resources. Wity’s accuracy was close to Prism’s, though Prism came out ahead on their evals. The task: speech-to-text still mishears learners, especially Indian-accented, mixed-language speech. In their data, 148 of 1,686 learner turns (about 9%) had a mishearing that changed the tutor’s reply. The root post is text only.
Original posthttps://x.com/maharishi_r_b/status/2121115692871356831