Does AI Give Different Answers Each Time? We Tested Ours

“AI gives you different answers each time.” I’ve heard this in different forms many times, and for AI analytics it’s a fair worry. A ruler that changes length every time you pick it up is useless. So we tested ours like a ruler.

The test

We took Red Dead Redemption 2 and drew four independent samples of its reviews: 2,000, 5,000, 10,000 and 20,000 reviews. Same date range, but pulled at random, so any two samples shared only 1–5% of their reviews.

Each sample ran through the Ludopsy pipeline, which scores 50+ topics on how often players bring them up and how strongly they feel about them, alongside the qualitative analysis.

If the measurement holds still, four different sets of reviews from the same period should give the same reading of the game.

The result: the same reading, four times

They did. That’s because we’ve approached the pipeline as measurement development from the beginning.

• How often a topic comes up landed within 1.4 percentage points across all four samples.
• How players feel about it held just as steady. For any topic mentioned by more than 2% of players, the median gap between scores on the 1–7 scale was under 0.2 points.

Where the results move, and why

The readings move exactly where you’d expect: rare topics. A topic raised by around 1% of players rests on a few dozen reviews or fewer, so its intensity score can fluctuate, the same way a survey of 20 people is shakier than a survey of 2,000.

Why it matters

Stable measurement is more than reassurance. An analysis that doesn’t shift with the sample is one you can base decisions on: you can benchmark and compare games, and track a game over time and trust that the changes you see are real.

In short, you can make calls on the results without wondering whether the numbers would change if you ran the analysis again. Stable measurement is the foundation everything else in Ludopsy sits on.