People predict their own movie ratings better than recommendation systems

Personal expectations about future movie ratings are more accurate than models built from the behavior of many people.

People predict how they will rate movies better than recommender systems. These systems usually learn what people want from what they have done.

Researchers used the MovieLens Beliefs dataset from 2023-2024. A homepage prompt asked members how they expected to rate movies they had not seen. Out of 26,020 expectations, 551 from 277 members were followed by a rating at least a day later.

The study compared expectations against four benchmarks. These included the platform's own predicted rating and two collaborative-filtering models. Collaborative-filtering models are matrix factorization and an item neighborhood. A stack trained on other members' outcomes was also used.

The correlation for people was 0.69. This was higher than 0.34 for MovieLens and 0.54 to 0.59 for the trained recommenders. Adding the expectation lowers out-of-sample mean squared error by 26%.

The results reconcile two theories. The crowd's data suffice where they are rich. A person adds information about a specific item.

Key facts

  • People's rating predictions had a correlation of 0.69.
  • The trained recommenders had correlations between 0.54 and 0.59.
  • Adding expectations lowers out-of-sample mean squared error by 26%.

All sections

Edition