alternative rankings on Analysis Paralysis

Debiasing the BoardGameGeek ranking

contact@recommend.games (Recommend.Games) — Fri, 20 Sep 2024 22:34:11 +0300

Bias is a bit of an ugly word, isn’t it? It certainly has become one of those battle phrases in the culture war, where both sides of the argument accuse the other of forcing their biases onto society. Board game reviews frequently need to justify themselves for their biases affecting their views. Dan Thurot recently wrote a very eloquent piece on the matter, diving deeper into different kinds of biases.

But bias also has a well defined meaning in statistics. Moving from emotions to cold hard numbers, the word bias loses its antagonistic nature and simply becomes a measurement one might want to minimise or remove entirely. Hence, debiasing the BoardGameGeek (BGG) ranking is about asking the question what it would look like if we removed the influence of a particular parameter. One such parameter is a game’s age: we’ve seen in the previous article that ratings have gone up over time, so removing the age bias from the BGG ranking means correcting for this trend.

This is by no means a new idea: Dinesh Vatvani published an often referenced article back in 2018 focussing on removing the complexity bias from the ratings. This article is an update to and an extension of his work.

Removing the age bias

So, let’s start with the bias readers of this blog will already be familar with: age bias, which really is just a slightly more neutral term for cult of the new. The first step is plotting the games’ ages vs their average ratings:

The points on the very left are the oldest games in our dataset, those published in 1970, whilst the ones on the right are those published in 2024. I’ve plotted games with few ratings more faintly in order to declutter the image. The bold red line is the line of best fit, i.e., the trend line that best describes the yearly increase in average ratings. This picture should look pretty familiar if you’ve read the previous article, though here we plotted every single game as an individual dot instead of aggregating by year.

That trend line has a slope of 0.03, which means that overall, a game’s average rating decrease by 0.03 with every year that has passed since its release. Now, removing the age bias means reducing that slope to 0. It’s as if we consider each year on its own and only care how much better or worse a game was compared to its peers released at the same time. I hope this little animation will make things much clearer:

(Again, credit to Dinesh Vatvani for introducing this kind of visualisation in his article.)

Next, we can use those adjusted average ratings to calculate a new, debiased ranking. In order to do this, we recreate the BGG ranking by taking the Bayesian average, i.e., adding 2311 dummy ratings (one for every 10,000 ratings in total) of 5.5.

Without further ado, those are the new top 10 rated games after removing the age bias:

Rank	Game	Rating
#1 (🔺 2)	Gloomhaven (2017)	8.4 (🔻 0.2)
#2 (🔺 11)	Twilight Struggle (2005)	8.4 (🔺 0.2)
#3 (🔻 1)	Pandemic Legacy: Season 1 (2015)	8.4 (🔻 0.1)
#4 (🔺 4)	War of the Ring: Second Edition (2011)	8.5 (🔸 0.0)
#5 (🔻 4)	Brass: Birmingham (2018)	8.4 (🔻 0.2)
#6 (🔺 145)	Sherlock Holmes Consulting Detective (1982)	8.5 (🔺 0.8)
#7 (🔸 0)	Terraforming Mars (2016)	8.2 (🔻 0.2)
#8 (🔻 4)	Ark Nova (2021)	8.2 (🔻 0.3)
#9 (🔺 36)	Puerto Rico (2002)	8.2 (🔺 0.2)
#10 (🔻 1)	Star Wars: Rebellion (2016)	8.2 (🔻 0.2)

As designed, older games are the big winners of this adjustment, with former BGG #1s Twilight Struggle and Puerto Rico making a comeback. Classic murder mystery game Sherlock Holmes Consulting Detective makes a huge leap into the top 10 as one of the few games from the 80s that stood the test of time. You can download the full new ranking here.

Removing the complexity bias

Obviously, we can apply the exact same idea to other features, e.g., a game’s complexity (or weight) as Dinesh Vatvani did in his original article. Again, we start by looking at the spread of the data points, from the lightest games on the left to the heaviest on the right:

The trend line has a slope of 0.63, i.e., the heaviest games on BGG have on average a full 2.5 point higher average score than the lighest ones. Somehow it feels particularly pleasing to see the light but clever games being lifted up when we remove this complexity bias:

This is the new top 10 after adjusting for the complexity bias:

Rank	Game	Rating
#1 (🔺 141)	Just One (2018)	8.2 (🔺 0.6)
#2 (🔺 127)	Codenames (2015)	8.0 (🔺 0.4)
#3 (🔺 45)	Cascadia (2021)	8.0 (🔺 0.1)
#4 (🔺 101)	SCOUT (2019)	8.2 (🔺 0.4)
#5 (🔻 3)	Pandemic Legacy: Season 1 (2015)	8.0 (🔻 0.6)
#6 (🔺 13)	7 Wonders Duel (2015)	7.9 (🔻 0.2)
#7 (🔺 30)	The Crew: Mission Deep Sea (2021)	8.1 (🔻 0.1)
#8 (🔺 71)	Azul (2017)	7.8 (🔺 0.1)
#9 (🔺 117)	Patchwork (2014)	7.8 (🔺 0.2)
#10 (🔺 57)	The Quacks of Quedlinburg (2018)	7.8 (🔸 0.0)

I’m not going to lie: As a lover of small and interactive games, this top 10 looks much more apealing to me than the actual BGG top 10. One striking observation: six out of these ten games won Spiel or Kennerspiel des Jahres (if we generously count the other Crew’s title), with two more nominated and the other two recommended. The grognards on the internet can complain all they want about how the awards have become too shallow, but the jury really knows how to pick out the games that put the maximum amount of game play into minimal rules. You can download the full new ranking here.

Removing the playing time bias

Next, we’ll take a look at how a game’s playing time (as measured by the minimum playing time printed on the box) affects its rating:

We see a similar trend line as with the complexity bias, which shouldn’t come as a surprise since a game’s length and complexity are highly correlated. The slope here is 0.0045, i.e., every minute it takes longer to play a game makes it 0.0045 points “better”, which does sound a bit weird when one puts it like that.

Because publishers love to lie about playing time and claim most games can be played in about an hour, the adjusted ranking doesn’t look all that different from the usual one, so I’ll skip it for this article, but you can download it here if you’d like to take a look anyways.

Removing the game type bias

So far, we’ve been looking at continuous values (also known as numerical features), but we can apply the same principal to categories. One particularly interesting feature in this context is a game’s type: BGG maintains separate rankings for eight different game types, and users can vote which of those types a game belongs to (depending on the share of votes, a game might be classified as more than one type). Given everything we’ve seen so far, you can probably already guess that the lighter categories like children’s and party games aren’t as highly praised as war and strategy games. Here’s all eight types sorted from lowest to highest average rating:

Each of those little violins represent the distribution of ratings amongst that type, with the white line indicating the median. Removing the game type bias now means bringing those distributions in line:

Just like before, we can calculate a new, debiased ranking:

Rank	Game	Rating
#1 (🔸 0)	Brass: Birmingham (2018)	8.2 (🔻 0.4)
#2 (🔺 7)	Star Wars: Rebellion (2016)	8.2 (🔻 0.2)
#3 (🔺 1)	Ark Nova (2021)	8.1 (🔻 0.4)
#4 (🔺 36)	Marvel Champions: The Card Game (2019)	8.2 (🔺 0.1)
#5 (🔺 43)	Cascadia (2021)	8.1 (🔺 0.1)
#6 (🔺 31)	The Crew: Mission Deep Sea (2021)	8.2 (🔺 0.1)
#7 (🔺 34)	Heat: Pedal to the Metal (2022)	8.1 (🔺 0.1)
#8 (🔻 2)	Dune: Imperium (2020)	8.0 (🔻 0.4)
#9 (🔻 2)	Terraforming Mars (2016)	7.9 (🔻 0.4)
#10 (🔺 11)	Nemesis (2018)	8.0 (🔻 0.2)

At first glance, it might seem curious that some games from the current BGG top 10 stay (more or less) put whilst others fall off. The reason for this is that both Pandemic Legacy: Season 1 and Gloomhaven are considered to be both strategy and thematic games, two of the most popular categories, and so the model weighs them down twice. Instead, we see some of the top rated customisable, abstract and family games in this adjusted top 10. Again, you can download the full new ranking here.

Removing the bias towards cooperative games

Let’s take a look at one final categorical feature to debias: competitive vs cooperative games. Cooperative games have been on the rise for a number of years now, winning six out of ten Spiel and Kennerspiel des Jahres awards since I started my predictions in 2020. While less than 9% of all games in our dataset are cooperative, on average they have a significantly higher (by almost 0.48 points) rating compared to competitive games:

We can go through the exercise of debiasing the ratings and calculating a new ranking here too, but the outcome would pretty much be just the standard BGG ranking with the cooperative games filter out (or rather weighed down). If you really want to take a look, you can download the new ranking here.

Removing all the biases

OK, so you might be wondering by now why I went through all that trouble, in particular since complexity, playing time and game types are all so strongly correlated. You’ve probably also been thinking association isn’t causation. You’d be right: viewing those different features individually, this approach yields nothing but correlations. But take them all together and we get a shot at a bit of causal inference.

How does it work? So far, I’ve calculated those trend lines using simple linear regression (also known as the ordinary least squares method) in a single explanatory variable. But the maths works just the same in higher dimensions and we can throw all the features discussed above at once into a single model, predicting again the game’s rating, but now with much more information. The outcome is this:

Estimated rating =
-0.031 * age in years
+0.567 * complexity score
-0.001 * minimum playing time in minutes
+0.167 if an abstract game
-0.029 if a children’s game
-0.030 if a customizable game
+0.225 if a family game
+0.274 if a party game
+0.125 if a strategy game
+0.125 if a thematic game
+0.485 if a war game
+0.199 if cooperative
+5.700

It’s worth taking a look at and comparing some of those coefficients. First of: Age and complexity have about the same influence in this combined model as they had individually. This means that age and complexity biases are real. Even when taking other factors into account, gamers clearly strongly prefer newer over older and heavier over lighter games.

Interestingly, something different is happening with playing time: If you recall, the original model estimated that every minute of additional playing time increased the rating by around 0.005, but this model tells us that every minute extra actually decreases a game’s rating by 0.001. This number is very small, but the direction is still statistically significant. This is a sign that the model correctly decoupled what we discussed before intuitively: the positive correlation between a game’s length and rating can be explained by the game’s complexity. Once we take that into account, any additional playing time actually harms the game’s rating. In other words: Our model finds that – all other features being equal – players do prefer shorter over longer games.

We observe some similar effects with the game types. In the single feature model, children’s games were rated over 0.6 points lower. But in this multi feature model, the coefficient is very close to zero, in fact not statistically significantly different. This is evidence that the ‘geeks don’t actually have a systematic bias against children’s games – they dislike all light games equally, no matter whether they are for children or adults.

Finally, the bias in favour of cooperative games is still pretty clear, but at around 0.2 it’s less than have what the single feature model predicted. They clearly are very fashionable, but much of their higher average ratings can be explained by other variables.

Enough looking at coefficients, let’s finally move on to the moment you didn’t even know you were looking forward to: the ultimate debiased ranking. Just like with all the other models, we can use this multivariate model to debias the games’ ratings by simply setting all those coefficients to zero. We lose the ability to animate this process for easy visualisation, but we gain average ratings and a ranking freed of all the biases we’ve been talking about:

Rank	Game	Rating
#1 (🔺 323)	Survive: Escape from Atlantis! (1982)	8.2 (🔺 0.9)
#2 (🔺 772)	Can’t Stop (1980)	8.2 (🔺 1.3)
#3 (🔺 326)	For Sale (1997)	7.9 (🔺 0.7)
#4 (🔺 15)	7 Wonders Duel (2015)	7.8 (🔻 0.3)
#5 (🔺 146)	Sherlock Holmes Consulting Detective (1982)	7.9 (🔺 0.3)
#6 (🔺 209)	Tichu (1991)	8.0 (🔺 0.4)
#7 (🔺 205)	Modern Art (1992)	7.9 (🔺 0.4)
#8 (🔺 637)	Take 5 (1994)	7.8 (🔺 0.8)
#9 (🔺 316)	Lost Cities (1999)	7.7 (🔺 0.5)
#10 (🔺 204)	Carcassonne (2000)	7.6 (🔺 0.2)
#11 (🔺 177)	Jaipur (2009)	7.7 (🔺 0.2)
#12 (🔺 117)	Codenames (2015)	7.6 (🔺 0.1)
#13 (🔺 125)	Ra (1999)	7.7 (🔺 0.1)
#14 (🔺 141)	Ticket to Ride: Europe (2005)	7.6 (🔺 0.1)
#15 (🔺 76)	El Grande (1995)	7.7 (🔻 0.1)
#16 (🔻 9)	Terraforming Mars (2016)	7.6 (🔻 0.8)
#17 (🔻 15)	Pandemic Legacy: Season 1 (2015)	7.6 (🔻 0.9)
#18 (🔺 289)	Telestrations (2009)	7.8 (🔺 0.4)
#19 (🔺 211)	Ticket to Ride (2004)	7.5 (🔺 0.1)
#20 (🔺 85)	SCOUT (2019)	7.8 (🔻 0.1)
#21 (🔺 273)	Battle Line (2000)	7.8 (🔺 0.3)
#22 (🔺 634)	HeroQuest (1989)	7.8 (🔺 0.6)
#23 (🔺 368)	Dixit (2008)	7.5 (🔺 0.3)
#24 (🔻 8)	The Castles of Burgundy (2011)	7.5 (🔻 0.6)
#25 (🔻 19)	Dune: Imperium (2020)	7.5 (🔻 0.9)

I will say: Survive: Escape from Atlantis! is not a game I would have expected on top of this list. In general, this top 25 is clearly much lighter and less recent than the regular BGG ranking. Personally, I’m up for this: it’s a wonderful list full of proven classics, devoid of any cult of the new or Kickstarter bloat.

Some games made huge leaps into the top 100, including Rummikub, Loopin’ Louie and Animal Upon Animal which all jumped up over 1000 positions. The biggest losers of this method – games dropping out of the top 100 – are On Mars, Root and Mage Knight, all of which lost 1000 positions and more. You can download the full debiased ranking here.

Some boring technical details

In order to have good data and comparable values for all those corrections, I had to filter out games by certain criteria. We only considered games which:

are ranked (i.e., rated by at least 30 users),
have a complexity score,
have been released between 1970 and 2024 and
have a minimum playing time of at most 3 hours.

This includes 23,325 of the 26,266 currently ranked games (88.8%), but does exclude some notable games, such as:

Twilight Imperium, Paths of Glory and Die Macher (too long), as well as
Crokinole, Go and Acquire (too old).

Those are the highest ranked exclusions. While it’s definitely sad to miss out on those and some other games, they make up only 2% of the top 1000 games on BGG, so I feel it’s a reasonable tradeoff.

Conclusion

So that’s a wrap on our quest for debiased rankings. It’s important to remember that bias isn’t a negative term in this context – we’re simply interested in the tendencies that pertain to the BGG user ratings and try to imagine counterfactual rankings if those tendencies didn’t exist. By considering not just a single feature like age or complexity, but throwing those two as well as playing time, game type and cooperative into one model, we hope to explain the way those features influence ratings beyond mere correlations. If you’re interested in the mathematical background to this method, I highly recommend the article I’ve already linked above.

As always, you can find the full analysis code and notebooks on GitLab.

The world of board games

contact@recommend.games (Recommend.Games) — Thu, 25 Apr 2024 23:02:24 +0300

BoardGameGeek (BGG) users can select their country of residence in their profile. The main purpose is to find other users in your region to play face to face or maybe trade games, but but over here at Recommend.Games we obviously cannot help ourselves but to use this information for some interesting statistics. 🤓

Let’s start with the usual disclaimer: We will have to rely on whatever information BGG provides. In particular, users can freely choose their country. As mentioned, this is meant to be the country of residence, but some users might rather choose their country of origin – or some outright nonsense. There are 19 BGG users who claim to be from 🇦🇶 Antarctica, for example. While I’m sure that the long polar nights on lonely research stations are perfect for playing board games, I’m not sure if they have a lot of time to rate them on BGG. 🐧 (If they actually did, Antarctica would have 6,880 ratings per 100 thousand residents, which would make them the second biggest board game enthusiasts in the world – only behind the 🇻🇦 Vatican’s 6 reported users, resulting in almost 20,000 ratings per 100 thousand residents. 🙏)

On a more serious note, what exactly is a country is a very political question and frequently highly controversial. I’m trying to stick with the sovereign nations recognized by the United Nations and️ add users from subdivisions and autonomous regions to the parent country. BGG allows users to choose 🏴󠁧󠁢󠁥󠁮󠁧󠁿 England, 🏴󠁧󠁢󠁳󠁣󠁴󠁿 Scotland or 🏴󠁧󠁢󠁷󠁬󠁳󠁿 Wales as their residence (not Northern Ireland though 🤔) – but for now, they’re all stuck together in a 🇬🇧 United Kingdom, if they like it or not. The borders in the map that follows shortly are even more contentious – I just had to stick with what the provider chose, and they emphasised the actual situation on the ground.

With that out of the way, let’s look at some basic statistics. For this analysis, we used 19.8 million ratings by 347 thousand users who selected one of the 199 countries as their residence. Close to 88 thousand games received at least one rating.

In order to understand how board game preferences differ around the globe, we want to calculate a ranking similar to the way BGG ranks games. Remember that BGG calculates their “geek score” based on a Bayesian average by adding a number of dummy ratings of 5.5 to each game. That number of dummy ratings is determined by the total number of ratings divided by 10,000. We can apply exactly the same logic to calculate a ranking by country. Since we want to apply at least one dummy rating to each game, we only consider countries with at least 10,000 ratings in total – 62 countries meet this criterion. BGG ranks games with at least 30 ratings. If we applied the same rule to the country rankings, we would often have very short lists. So, I decided to include games with at least 3 times the number of dummy ratings in the country ranking, but never more than the BGG standard of 30.

Enough of the boring technicalities, on to the fun part! 😎 Let’s first answer some basic questions: How many users from that country? How many ratings have the entered? How many ratings per capita? And maybe most interestingly: What is their favourite game? Here’s a quick overview over the 10 largest communities on BGG:

Country	Population	Users	Ratings	Per 100k residents	#1 rated game
🇺🇸 United States	331.9 mil	131.7k	7,724k	2,327	Pandemic Legacy: Season 1
🇩🇪 Germany	83.2 mil	19.0k	1,457k	1,751	Ark Nova
🇨🇦 Canada	38.2 mil	23.4k	1,390k	3,636	Gloomhaven
🇬🇧 United Kingdom	67.3 mil	24.3k	1,300k	1,931	Pandemic Legacy: Season 1
🇪🇸 Spain	47.4 mil	16.0k	913k	1,926	Brass: Birmingham
🇵🇱 Poland	37.7 mil	11.8k	620k	1,644	Star Wars: Rebellion
🇮🇹 Italy	59.1 mil	9.1k	550k	930	Through the Ages: A New Story of Civilization
🇫🇷 France	67.7 mil	10.6k	529k	782	Ark Nova
🇳🇱 Netherlands	17.5 mil	7.5k	469k	2,680	Brass: Birmingham
🇦🇺 Australia	25.7 mil	9.2k	468k	1,824	Pandemic Legacy: Season 1

We see lots of familiar names amongst the highest rated games. The 62 country rankings have 26 different games in the top spot. These nine games are favourites in more than one country:

Game	Count
Brass: Birmingham	14
Gloomhaven	8
Pandemic Legacy: Season 1	6
Ark Nova	5
Twilight Imperium: Fourth Edition	4
Terraforming Mars	3
War of the Ring: Second Edition	2
Star Wars: Rebellion	2
Gloomhaven: Jaws of the Lion	2

This looks almost identical to the BGG top 10 list – only Dune: Imperium is missing, though it is 🇦🇷 Argentina’s favourite game. Overall, the country rankings are dominated by international hits, but there are a couple of local favourites with a strong tie to the country, such as Go in 🇨🇳 China and AFU: Armed Forces of Ukraine in 🇺🇦 Ukraine.

Explore more details about the different countries in this interactive map:

As much fun as it is to browse the map, there’s also the blank spaces staring in our faces: Africa and South Asia, together representing over 40% of the world’s population, are almost completely missing from the international board gaming community. 🌍 Obviously, BGG being an American website only available in English means their data really only represents the Anglo-American gaming world, but still, it’s notable how Latin America and other parts of Asia are present much stronger on BGG. I think some focus and effort needs to go into closing those gaps on the world map. 🕊️

I couldn’t possibly leave you without circling back to the question of what country is the most enthusiastic about board games. We’ve already mentioned the statistical anomalities that are the 🇻🇦 Vatican and 🇦🇶 Antarctica, but if we only consider the 62 countries with their own rankings, which of those has the most ratings per capita? Here’s the top 5:

🇫🇮 Finland (4953 per 100k residents)
🇮🇸 Iceland (4932 per 100k residents)
🇨🇦 Canada (3636 per 100k residents)
🇧🇪 Belgium (3575 per 100k residents)
🇸🇪 Sweden (3337 per 100k residents)

Those who were looking for the reason why my chosen home is the happiest country in the world, you’ve finally got the answers: we seem to play more board games than anyone else! (Torille! 🇫🇮) In general, the Nordic countries sure hit the sweet spot here, with their highly educated populations (well versed in English) and strong emphasis on work–life balance. Also, you gotta have something to do during those long, dark winter nights. 🌌

Finally, if you’re interested in the full country overview, you can find it here. Browse the full rankings for each country here:

PS: As always, the complete code for this analysis can be found from GitLab.

Recommend.Games ranking explained

contact@recommend.games (Recommend.Games) — Tue, 22 Feb 2022 22:02:22 +0200

We have some pretty exciting news! 🤩 Since 2022-02-22, we’ve been using a new and improved ranking as the default sorting for our front page at Recommend.Games.

… What do you mean, you haven’t noticed there’s a R.G ranking? You know, that thing that’s referenced in the statistics? … You haven’t ever opened that view?!? 😱

OK, let’s back up for a second. R.G’s primary purpose is a recommendation engine, that is, finding personalised game recommendations based on a user’s preferences (in this case as expressed through their BoardGameGeek ratings). But we want to show some games to the users immediately when they load the start page, before entering their user name. Think of these games as recommendations for… anyone. What games would you recommend to a person if you didn’t know anything about their taste? This is the implicit purpose of any “Top X entities” list. Since this is a recommendation site, we’d like to make this claim explicit, so the R.G ranking has always been those “recommendations for everyone”. What’s changed is the way those recommendations are calculated.

Recommendations are based on a technique known as collaborative filtering. Basically, the algorithm tries to learn the users’ preferences based on their ratings, find users with similar tastes, and then recommend new games that user might like. R.G uses Apple’s implementation Turi Create for its recommendations, which does a lot of magic 🪄 under the hood. One of those magic tricks is that it offers recommendations for users it doesn’t know anything about. The algorithm treats those users as new and offers some default recommendations that should be a good starting point for anybody, without any particular context. It’s exactly those recommendations that R.G has been using for years as our ranking and default sorting.

So, why change this and create a new ranking? There’s a number of problems with the old ranking. For one, it’s really swingy.¹ Seriously, just take a look at the history:²

Maybe more importantly, the exact algorithm to determine those recommendations for new users is extremely obscure. There’s been a long-standing ticket to find out what’s going on. The answer isn’t documented anywhere, but you can find it somewhere in these lines of code. Let me know if you can make sense of them, I’ve simply given up at some point.

Instead, I wanted to create a new ranking which I can at least explain. So here goes:

The basic idea is to calculate the recommended games for all users, and then average those rankings. Those averages would become the Recommend.Games rankings.

That sounds simple and intuitive, but, as the saying goes, the devil is in the details. For starters, there are over 100,000 games in the BoardGameGeek database, and over 400,000 users with at least one rated game. Calculating all recommendations for all users would therefore mean over 40 billion user–game pairs. That’s a lot. 😅 But really, we don’t care if a game is the 1,000th, 10,000th or 100,000th highest rated game. Instead, we only recommend the top 100 games for each user. The highest game on that list receives 100 points, the next 99, and so on, until the 100th game receives a single point from that user. All other games will be awarded 0 points. The we can simply average those points across all users, and voilà, those scores become the R.G rankings. 🤩

… Except that’s still not the whole story. I wasn’t happy about the idea of a user who fifteen years ago left a single rating being equally important as someone who played and rated hundreds of board games over two decades of BGG history. That’s why I tried to model the “trust” we should put in different users’ ratings. The idea here would be that we should trust a user’s rating more if two things are true:

They are a regular contributor to BGG, i.e., rate, log plays etc at least once a month over a long period of time.
They should rate games with a reasonable distribution, i.e., on a nice bell curve.

For the first point we count each calendar month that the user was active on BGG (e.g., rated a game, logged a play, updated their collection). Then we take the logarithm (base 2) of that number – this means the score will rise quite rapidly in the first year of BGG activity, but then flatten out quite quickly. The goal is not to treat newcomers too harshly. Note that \(\log_2 1 = 0\), so new users with only one month of activity will start with zero trust and consequently not be considered in the rankings calculations.

The second point is a little trickier. Mathematically speaking, we want to assess how closely a user’s ratings follow a normal distribution. There are a couple of neat statistical tests that do that, the one that’s considered best is the Shapiro–Wilk test. What we care about is a number between 0 and 1 that somehow measures how much a user’s rating distribution resembles a bell. Take, for instance, Tom Vasel’s ratings. He clearly users a nice spread for his ratings:

If on the other hand a user only ever rates games with a 10, never any other score, their factor would be 0, and again, they wouldn’t take part in the R.G rankings calculations.

So, long story short, we have those two factors that are supposed to describe how much we can trust a user’s ratings. We can now multiply them with each other, and the higher the score, the higher will be our trust in their ratings, and consequently, the more weight their “vote” will have when calculating the R.G ranking.

But before we go there, what users do we trust the most? Many of the highest scores actually belong to pretty random users, so I’m not comfortable exposing them here, but it’s interesting to compare some of the “celebrities” in the hobby (and yours truly):

User	Rank	Trust
W Eric Martin (BGG news)	21	7.376
Engelstein (Ludology)	77	7.227
Tomvasel (Dice Tower)	312	7.009
Aldie (BGG founder)	1889	6.544
Jameystegmaier (Stonemaier)	3213	6.343
Jonpurkis (Actualol)	3628	6.296
Bohnanzar (Uwe Rosenberg)	8801	5.848
Markus Shepherd 🤓	16207	5.453
Quinns (SU&SD)	16958	5.421
Phelddagrif (Richard Garfield)	67960	3.981
Cephalofair (Isaac Childres)	75010	3.833
Elizharg (Elizabeth Hargrave)	108248	3.156

My opinions on board games having more weight than Quintin Smith’s sure feels good. 🧐

In case you’re curious: The reason why the recommendations are so swingy is because they aren’t precisely calculated, but merely approximated by an algorithm called stochastic gradient descent, which is inherently non-deterministic. ↩︎
And this is even a smoothed version of the rankings: It uses the average score of one week to determine the ranking in that plot. ↩︎

Reverse engineering the BoardGameGeek ranking

contact@recommend.games (Recommend.Games) — Sat, 03 Oct 2020 08:42:51 +0300

TL;DR: BoardGameGeek calculates its ranking by adding around 1500-1600 dummy ratings of 5.5 to the regular users’ ratings. They called it their geek score, statisticians call it a Bayesian average. We use this knowledge to calculate some alternative rankings.

I often describe BoardGameGeek (BGG) as “the Internet Movie Database (IMDb) for games”. Much like its cinematic counterpart, the biggest board game database not only collects all sorts of information obsessively, but also allows users to rate games on a scale from 1 (awful - defies game description) to 10 (outstanding - will always enjoy playing). These ratings are then used to rank games, with Gloomhaven occupying the top spot since December 2017.

While BGG founder Scott Alden admitted in a recent interview on the excellent Five Games For Doomsday podcast that he doesn’t care all that much about the rankings, gamers around the world certainly do. They would discuss heatedly any movement in the rankings, question why games X is up there while game Y is missing, and generally criticise the selection for either having too many or not enough recent releases.

Reason enough for me to take a closer look at how the rankings work and some of the maths behind it.

Generally speaking, we want to rank a game higher the better its score is. The first instinct would be to just sum up all the ratings users gave to that particular game, divide by the number of ratings, and rank games from highest to lowest. What I just described would be the arithmetic mean (or just average if you feel less fancy) of the ratings, which is simple and intuitive, but suffers from a sever defect: a game with a single rating of 10 would always sit on top of the ranking, well ahead of much beloved games with thousands of votes that couldn’t possibly be all 10s.

The easiest fix is filtering out games with less than a certain number of ratings, say 100.¹ That’s a decent enough approach, and yields the following top 5 games as of the time of writing:

Notably, those are all very recent games with relatively few ratings.² Some might consider this a feature, not a bug, but when your intention is to create a list of the best board games, you probably do want to give a nod to proven classics, and not just the latest hotness. How to balance out these ends of the spectrum is in the end a choice you have to make, and no matter what it is, People on the Internet™ will not like it.

The way both IMDb and BGG chose to tackle this issue is by essentially not trusting the ratings – at least not too much. The method boils down to assigning a new item in the database (be it movie or game) a predefined average by default, and only gradually trusting the ratings’ average as thousands and thousands of users have cast their votes. More concretely the rankings are calculated by adding a number of dummy ratings with a chosen average value, say 5.5, to each game’s regular ratings. The result is that initially each game will have a score close to 5.5, but as more users rate the game, that score will move closer and closer to the conventional mean.

BGG calls this their geek score. Mathematically speaking, it is a Bayesian average, and calculates as follows:

\[ \textrm{geek score} = \frac{\textrm{sum of ratings} + \textrm{number of dummies} \cdot \textrm{dummy value}}{\textrm{number of ratings} + \textrm{number of dummies}}, \]

where sum of ratings can be calculated either by, well, summing up all ratings or via number of ratings ⋅ average rating. Don’t worry too much about the details though – adding dummy ratings is really all you need to understand.

OK, so that’s the concept, but crucially that’s not all the details. You still need to choose how many dummy ratings you want to add and what value they should take. Since People on the Internet™ who disagree with your ranking will try to manipulate it in whatever way they can, sites are usually very cagey about said details. IMDb used to be more transparent, as was BGG, but now we have to dig a little deeper.

Let’s start from the easier of the two, the value of the dummy ratings. It is commonly chosen to represent some prior mean, i.e., some decent estimate of the rating a new game in the database would have. A frequent choice would be to use the average rating across all games. It’s a fair assumption – without further information about a game, we don’t know if it’s any better or worse than the average game. However, Scott Alden actually gave away the answer in that interview from the beginning: BGG chose the dummy value to be 5.5. Their rationale is that ratings range from 1 through 10, so 5.5 is the midpoint. Of course, people tend to rather play and rate much more the games they like, and so the average rating is around 7. Opting for the lower value here is part of the design of the ranking: it means a new game would enter the ranking rather at the end of the pack. On the other hand, using the mean as the dummy value means a new game is placed more or less in the middle. It is worth mentioning that IMDb does use the mean (or at least used to), but they only ever publish the top 250 movies, and don’t care about the crowd behind.

The other value, the number of dummy ratings, requires more work. Because some of the details and data are unknown, we cannot actually pin down the exact number that BGG is using. Instead, we’ll try three different approaches, and compare their results.

Formula

On the surface, this should be super easy to solve: in the formula above, we know every single value but the number of dummy ratings. BGG publishes the number of ratings, their arithmetic mean, and the “geek score” or Bayesian average for every game, and we know that the dummy value is 5.5. With a little high school algebra we solve the formula for number of dummies:

\[ \textrm{number of dummies} = \textrm{number of ratings} \cdot \frac{\textrm{average rating} - \textrm{geek score}}{\textrm{geek score} - \textrm{dummy value}} \]

Now we should be able to plug in those values for any given game, say Flamme Rouge, and get the result. With 10,936 ratings that average 7.562, and a geek score of 7.266, this yield:

\[ \textrm{number of dummies} = 10936 \cdot \frac{7.562 - 7.266}{7.266 - 5.5} \approx 1830. \]

So, there’s about 1830 dummy ratings, end of story. Right? Unfortunately, not quite. When computing this formula for different games, the results vary wildly, as you can see from this histogram over the results for the same calculation with other games:

And this plot is even cropped, the results vary from -1.4 million to +810 thousand, though some 90% lie within the above range, with a mean of around 1604 and a median of around 1590.

What’s going on, why are the results so inconsistent? The problem is the ranking’s secret sauce. Both IMDb and BGG stress is that they only consider regular voters for their rankings. That’s the most mysterious part of the system as it’s the easiest to manipulate, so we’ll just have to take their word for it. For this investigation it means that the average rating BGG publishes includes all the ratings, but the geek score might not.

Still, clearly something is happening around the 1600 ratings mark, so we are at least getting closer to an answer. If exact calculations won’t work, maybe we can approximate the correct value instead?

Trial & error

Let’s take a step back here. What we’re really trying to achieve here is not finding the exact formula for that mysterious “geek score”, but rather recreate the BGG ranking. That is, we want to find the values in the above formula, such that the resulting ranking matches BGG’s ranking as closely as possible. Luckily, statistics has all the tools we need. Spearman correlation measures rank correlation – just what we need. This will be 1 if both rankings sort in exactly the same way, 0 if there’s no relation, and -1 if they sort exactly the opposite way. Again, don’t worry about the details, just trust the maths.

What we can do now is fairly simply and quickly compute the rankings for different number of dummy ratings, and pick the value with the highest Spearman correlation. Without further ado, here are the results:

The best correlation of around 0.996 is achieved with 1488 dummy ratings. However, it is worth noticing that the changes in the correlation are very, very small throughout the range we examined here (1000 to 2500), so let’s dig still a little deeper.

Optimisation

What we have here at hand is actually a classic optimisation task: a real valued function in one unknown (or two if we allow a variable dummy value as well) which we’d like to maximise. This is a well-studied field, with many fast and simple implementations that provide us the solution in no time. Unsuprisingly, we get the same result as above: the best possible correlation is 0.996 with around 1488 dummy ratings.

But since we made it this far, let’s take it one step further. So far, we tried to optimise the correlation in order to recreate BGG’s ranking. However, we can also try to recreate the actual geek scores. That is, we can look for the number of dummy ratings that will yield the closest to the actual geek score with our calculations. What exactly we mean by “closest” is up to us to define. A common metric is the mean squared error.³ It’s not worth getting into the maths here either, but the general idea is that we want to punish outliers in our estimates more (qudratically so) the further away they lie from the actual datapoint. Long story short, this yields a minimum for around 1636 dummy ratings.

Let’s take one last swing and see what happens if we don’t fix the dummy value at 5.5 but allow that to be variable as well. This is no problem for the optimisation algorithm and yields the following results:

the best correlation with 1942 dummy ratings of 5.554, and
the least squared error with 1616 dummy ratings of 5.494.

Either of those improvements in the performance metrics are hardly noticable (in fact insible after rounding), but they do confirm nicely a dummy value of 5.5.

Conclusion

All things consider, we can be confident that BoardGameGeek calculates their rankings by adding around 1500 to 1600 dummy ratings of 5.5 to the regular users’ ratings. What exactly constitutes a regular user, and what ratings might be discarded due to shilling, remains a well guarded secret though. Note that the number of dummies is pegged to the overall number of ratings, so this is a moving target, and the calculations would change as time passes.

Now I must applaud anybody who actually made it all the way through this pretty dry and technical article. The real reason why I dwelled so much on the ratings, and how they are compressed into the BGG rankings, is to get a feeling of what’s going on behind the scenes, what the can express, and what they cannot or even do not try to express. Another major take-away is that any of these decisions are choices that need to be made and that come with certain tradeoffs – like them or not.

In this particular case, I have the feeling that both the Cult of the New™ and connoisseurs of classic games are equally unhappy about the BGG top 100, which one should probably consider a compliment.

Alternative rankings

I’ll send you off with some rankings that were obtained by making different choices for the two values that we discussed throughout this article: the number of dummy ratings and their value.

Using the ratings average as dummy value

I’ve mentioned before that the average rating across all games is around 7 – a little¹ more precisely 7.08278. What if we chose that as the dummy rating, but left their number at 1600? The result should be a ranking that is a little friendlier to newer titles with fewer ratings as their score isn’t dragged all the way down to 5.5 in the beginning.

Sure enough, the brand new Jaws of the Lion with less than 3000 ratings already shows up in the top 10. The other game that sticks out here is Kingdom Death: Monster. This Kickstarter success story clearly attracted a lot of enthusiasts, but not necessarily the mass.

Using the top 250 number of ratings

Just like IMDb publishes only their top 250 movies, we can consider the same and crank up the number of dummy ratings. A good number seems to be the 250th most rated game on BGG, which has been rated 12,014 times. Using BGG’s standard dummy value of 5.5, we obtain a ranking that is much more skewed towards proven classics:

The most recent release on this list is Gloomhaven, but we also meet again old BGG #1’s: Puerto Rico and Agricola.

Combining both!

Finally, let’s do what IMDb does (or used to do), and add to each game’s ratings 12,014 dummy ratings of 7.08278:

The effects of more, but higher dummy ratings seem to almost cancel each other out. Compared to BGG’s actual top 10, only Twilight Imperium and Gaia Project are missing, otherwise this ranking looks very familiar. Turns out, BGG did a pretty good job designing its ranking!

Read the second part here where we nail down the number of dummy ratings to the dot.

PS: You can find the notebook I used to do all the calculations on Kaggle.

PPS: Turns out that GoDataDriven did almost the same calculation three years ago – even with the same title! Back then, they estimated that BGG added 725 dummy ratings. Jorge Nieva replicated their analysis with a more recent dataset, which thankfully yielded a very close match to our result: 1594 dummy ratings.

Throughout this article I only considered games with at least 100 ratings, mostly to ensure that the very long tail of games with few ratings won’t unduely skew the results. However, most of the calculations would only change in some negligible decimals when including all games. ↩︎ ↩︎
Jaws of the Lion is something of an exception here and will undoubtably shoot into the BGG top 10 very soon. In fact, it might be the only game with the potential to unseat Gloomhaven as the number 1. ↩︎
It’s probably even more common to use the root mean squared error, but for boring mathematical reasons, it doesn’t make a difference when it comes to optimisation. In fact, we could even drop the word mean from our metric and still obtain the same optimal point, so let’s not dwell on this. ↩︎