Asking for reviews changes how online ratings show information
New research shows that prompting users for reviews can increase positive scores and change how ratings react to new feedback.
Review prompts change what online ratings show. Review counts jump 4.25 times higher just above a specific threshold. The positive share of reviews grows by 6.6 percentage points.
Researchers studied 21.6 million Steam reviews of 315 games. Steam began prompting players for a review after about three hours of play in October 2019. They used a difference-in-discontinuities design to study this threshold.
Signal detection theory separates what a rating knows from what it shows. Sensitivity is what a rating knows. Criterion is what a rating shows. Selection theory implies that asking helps. Voice theory suggests it dilutes the news.
The movement of ratings changes significantly. Pass-through from reviews written in the first two hours falls from 0.73 to 0.18 just above the threshold. This is compared to 0.81 to 0.64 just below the threshold.
A fixed rating tier responds to 12% of downturns in reviews above the threshold. This is down from 42% compared to 41% and 47% below the threshold. Sensitivity falls more above the threshold because added reviews respond about half as much.
Review prompts mainly change how a rating must be read. Platforms that solicit reviews should recalibrate their rating labels.
Key facts
- Review counts become 4.25 times higher just above the three-hour threshold.
- The positive share of reviews grows by 6.6 percentage points after the prompt.
- Sensitivity changes by -0.03 relative to the stream below with a 95% confidence interval of -0.13 to 0.06.