I benchmarked our AI tool from 30% to 100% success

Show HN: I benchmarked our AI tool from 30% to 100% success

I benchmarked our AI tool from 30% to 100% success is a business in show hn launch based in Montral, Quebec, CA, founded in 2013.

Profile

Hi HN, I'm a Senior SDET at Plotly. My company just launched Plotly Studio, a new tool that uses AI to build data visualizations and analytics apps. My job was to answer the big question: does it actually work with real, messy data? When I first started testing it against our collection of 100+ diverse datasets, our success rate was around 30%. The problem I faced was that you can't just unit-test an AI that generates code for a desktop app. You have to test the full, end-to-end user experience. So, I led the effort to build our own internal benchmark system to validate performance at scale. Every day, our CI (GitHub Actions) kicks off a job that: Generates a full data app from each of our 1

Its activity falls under show hn launch; the profile below is assembled from public directory data.

The company is recorded as being based in Montral, Quebec, CA.

Its current recorded stage is Show HN launch.

The company reports a headcount in the 51-200 range.

Website: plotly.com/blog/chasing-nines-on-ai-reliability-benchmarks