@davepoon put this week’s Google Decisions API into his Trust Me Bro Phishing test, with the same questions and rules for every model: 200 real phishing emails, plus 200 normal emails written to look like them. Phishing caught / false alarms: TypeSafe Jev 180/8, OpenAI Decisions API 185/3, Cloudflare Clef 191/7, Gemini Decisions API 188/1. He says 1 false alarm in 200 is the lowest of any AI model he has tested, and that Gemini beat Jev on both counts by too wide a gap to be luck.
- 16 views
- 0 likes
- 0 bookmarks
- 0 reposts
About this case
Benchmark. davepoon (@davepoon) quotes Sundar Pichai (@sundarpichai) announcing the Gemini Decisions API. On his own numbers, Clef still catches a few more phishing emails, but he says that gap is too small to call. Downsides he names for Gemini: about 2x Jev’s price, and 709 ms per email from Sydney, of which around 350 ms is the trip to Google’s servers. He attaches his own video. The quoted launch video is the quoted post’s media.
Original posthttps://x.com/davepoon/status/2121390376447774858