Skip to content

@Mav1972 checked 54 storefront screenshots with short yes/no questions. Jev answered in 0.4 s a call, caught 19 of 22 planted defects, and raised 0 of 24 false alarms on harmless changes.

Benchmarks

Mav1972 @Mav1972

EN

  • 12 views
  • 0 likes
  • 0 bookmarks
  • 0 reposts
I checked 54 storefront screenshots with short yes/no questions about what each screen must show. Jev answered them in 0.4 s a call, caught 19 of 22 planted def…

About this case

Benchmark note. Mav1972 says they checked 54 storefront screenshots with short yes/no questions about what each screen must show. Jev answered them in 0.4 s a call, caught 19 of 22 planted defects, and raised 0 of 24 false alarms on harmless changes. One photo is attached. Its alt text calls it a bar chart of defects caught out of 22 and false alarms out of 24 for nine screenshot checkers, and names Jev with yes/no assertions at 19 caught and 0 false alarms, Gemini at 22 and 0, and pixel diff at 13 and 16.

Original posthttps://x.com/Mav1972/status/2117756513171485150

Related cases