Dummy Ratings on Analysis Paralysis

The world of board games

contact@recommend.games (Recommend.Games) — Thu, 25 Apr 2024 23:02:24 +0300

BoardGameGeek (BGG) users can select their country of residence in their profile. The main purpose is to find other users in your region to play face to face or maybe trade games, but but over here at Recommend.Games we obviously cannot help ourselves but to use this information for some interesting statistics. 🤓

Let’s start with the usual disclaimer: We will have to rely on whatever information BGG provides. In particular, users can freely choose their country. As mentioned, this is meant to be the country of residence, but some users might rather choose their country of origin – or some outright nonsense. There are 19 BGG users who claim to be from 🇦🇶 Antarctica, for example. While I’m sure that the long polar nights on lonely research stations are perfect for playing board games, I’m not sure if they have a lot of time to rate them on BGG. 🐧 (If they actually did, Antarctica would have 6,880 ratings per 100 thousand residents, which would make them the second biggest board game enthusiasts in the world – only behind the 🇻🇦 Vatican’s 6 reported users, resulting in almost 20,000 ratings per 100 thousand residents. 🙏)

On a more serious note, what exactly is a country is a very political question and frequently highly controversial. I’m trying to stick with the sovereign nations recognized by the United Nations and️ add users from subdivisions and autonomous regions to the parent country. BGG allows users to choose 🏴󠁧󠁢󠁥󠁮󠁧󠁿 England, 🏴󠁧󠁢󠁳󠁣󠁴󠁿 Scotland or 🏴󠁧󠁢󠁷󠁬󠁳󠁿 Wales as their residence (not Northern Ireland though 🤔) – but for now, they’re all stuck together in a 🇬🇧 United Kingdom, if they like it or not. The borders in the map that follows shortly are even more contentious – I just had to stick with what the provider chose, and they emphasised the actual situation on the ground.

With that out of the way, let’s look at some basic statistics. For this analysis, we used 19.8 million ratings by 347 thousand users who selected one of the 199 countries as their residence. Close to 88 thousand games received at least one rating.

In order to understand how board game preferences differ around the globe, we want to calculate a ranking similar to the way BGG ranks games. Remember that BGG calculates their “geek score” based on a Bayesian average by adding a number of dummy ratings of 5.5 to each game. That number of dummy ratings is determined by the total number of ratings divided by 10,000. We can apply exactly the same logic to calculate a ranking by country. Since we want to apply at least one dummy rating to each game, we only consider countries with at least 10,000 ratings in total – 62 countries meet this criterion. BGG ranks games with at least 30 ratings. If we applied the same rule to the country rankings, we would often have very short lists. So, I decided to include games with at least 3 times the number of dummy ratings in the country ranking, but never more than the BGG standard of 30.

Enough of the boring technicalities, on to the fun part! 😎 Let’s first answer some basic questions: How many users from that country? How many ratings have the entered? How many ratings per capita? And maybe most interestingly: What is their favourite game? Here’s a quick overview over the 10 largest communities on BGG:

Country	Population	Users	Ratings	Per 100k residents	#1 rated game
🇺🇸 United States	331.9 mil	131.7k	7,724k	2,327	Pandemic Legacy: Season 1
🇩🇪 Germany	83.2 mil	19.0k	1,457k	1,751	Ark Nova
🇨🇦 Canada	38.2 mil	23.4k	1,390k	3,636	Gloomhaven
🇬🇧 United Kingdom	67.3 mil	24.3k	1,300k	1,931	Pandemic Legacy: Season 1
🇪🇸 Spain	47.4 mil	16.0k	913k	1,926	Brass: Birmingham
🇵🇱 Poland	37.7 mil	11.8k	620k	1,644	Star Wars: Rebellion
🇮🇹 Italy	59.1 mil	9.1k	550k	930	Through the Ages: A New Story of Civilization
🇫🇷 France	67.7 mil	10.6k	529k	782	Ark Nova
🇳🇱 Netherlands	17.5 mil	7.5k	469k	2,680	Brass: Birmingham
🇦🇺 Australia	25.7 mil	9.2k	468k	1,824	Pandemic Legacy: Season 1

We see lots of familiar names amongst the highest rated games. The 62 country rankings have 26 different games in the top spot. These nine games are favourites in more than one country:

Game	Count
Brass: Birmingham	14
Gloomhaven	8
Pandemic Legacy: Season 1	6
Ark Nova	5
Twilight Imperium: Fourth Edition	4
Terraforming Mars	3
War of the Ring: Second Edition	2
Star Wars: Rebellion	2
Gloomhaven: Jaws of the Lion	2

This looks almost identical to the BGG top 10 list – only Dune: Imperium is missing, though it is 🇦🇷 Argentina’s favourite game. Overall, the country rankings are dominated by international hits, but there are a couple of local favourites with a strong tie to the country, such as Go in 🇨🇳 China and AFU: Armed Forces of Ukraine in 🇺🇦 Ukraine.

Explore more details about the different countries in this interactive map:

As much fun as it is to browse the map, there’s also the blank spaces staring in our faces: Africa and South Asia, together representing over 40% of the world’s population, are almost completely missing from the international board gaming community. 🌍 Obviously, BGG being an American website only available in English means their data really only represents the Anglo-American gaming world, but still, it’s notable how Latin America and other parts of Asia are present much stronger on BGG. I think some focus and effort needs to go into closing those gaps on the world map. 🕊️

I couldn’t possibly leave you without circling back to the question of what country is the most enthusiastic about board games. We’ve already mentioned the statistical anomalities that are the 🇻🇦 Vatican and 🇦🇶 Antarctica, but if we only consider the 62 countries with their own rankings, which of those has the most ratings per capita? Here’s the top 5:

🇫🇮 Finland (4953 per 100k residents)
🇮🇸 Iceland (4932 per 100k residents)
🇨🇦 Canada (3636 per 100k residents)
🇧🇪 Belgium (3575 per 100k residents)
🇸🇪 Sweden (3337 per 100k residents)

Those who were looking for the reason why my chosen home is the happiest country in the world 🗄️, you’ve finally got the answers: we seem to play more board games than anyone else! (Torille! 🇫🇮) In general, the Nordic countries sure hit the sweet spot here, with their highly educated populations (well versed in English) and strong emphasis on work–life balance. Also, you gotta have something to do during those long, dark winter nights. 🌌

Finally, if you’re interested in the full country overview, you can find it here. Browse the full rankings for each country here:

PS: As always, the complete code for this analysis can be found from GitLab.

Reverse engineering the BoardGameGeek ranking – Part 2!

contact@recommend.games (Recommend.Games) — Tue, 26 Jan 2021 22:24:00 +0200

This is the second part of a series explaining and analysing the BoardGameGeek rankings. Read the first part here.

Last time I left you with the nice result that BoardGameGeek (BGG) calculates its ranking by taking users’ ratings for a particular game and then add around 1500-1600 dummy ratings of 5.5. This so-called geek score is used to sort the games from best (Gloomhaven) to worst (Tic-Tac-Toe).

One detail however we touched on in passing, but did not resolve, is how that number of dummy ratings develop over time. When the current calculation method was introduced, BGG founder Scott Alden mentioned that this number would be pegged to the number of total ratings, but did not reveal any details. Challenge accepted!

In order to tackle this question, we need to compare that dummy number to the total number of ratings over time. Fortunately, thanks to the scraping done for Recommend.Games, we have access to the BGG games data over the past year or so. Using these snapshots, we observe how the number of games and ratings in the database has grown:

We can now repeat the exact same calculation we did in the previous post: For each point in time, the algorithm searches for the number of dummy ratings that yields an estimated geek score closest to the actual score. Now, we have a bunch of data points that correlate the total number of ratings with the number of dummies used at that time. Here’s what it looks like:

We get a pretty nice straight line – the dashed line in the plot is fitted with linear regression, i.e., the straight line that most closely fits our data. Its formula is:

\[ \textrm{number of dummies} \approx 0.0000997 \cdot \textrm{total number of ratings}. \]

This means that for every rating entered into the BGG database, the number of dummy ratings is increased by 0.0000997. That number might look a bit opaque, but it’s actually very easy to interpret once you put the question to its head: How many ratings have to be entered for the number of dummies to increase by 1? You get the answer to that by taking the inverse of that factor, which happens to be about 10,032. This number is way to close to 10,000 to be a coincidence! We can conclude the exact formula for the number of dummy ratings:

\[ \textrm{number of dummies} = \frac{\textrm{total number of ratings}}{10\,000}. \]

As of the time of writing, there are 17,287,904 ratings (give or take) in the BGG database, so there will be around 1729 dummy ratings of 5.5 added to the regular ratings.

As the number of BGG users rises steadily, this number of dummy ratings also keeps increasing. This is part of the reason why older games (particularly those with a newer edition) tend to drop in the rankings. When users stop adding new ratings, a game’s average rating more or less freezes. But because more and more dummy votes are added, the geek score decreases every time it gets recalculated, and so the older games drop in the rankings, while the latest hotness gets all the fresh votes, and shoots up to the top.

The circle of hype.

Reverse engineering the BoardGameGeek ranking

contact@recommend.games (Recommend.Games) — Sat, 03 Oct 2020 08:42:51 +0300

TL;DR: BoardGameGeek calculates its ranking by adding around 1500-1600 dummy ratings of 5.5 to the regular users’ ratings. They called it their geek score, statisticians call it a Bayesian average. We use this knowledge to calculate some alternative rankings.

I often describe BoardGameGeek (BGG) as “the Internet Movie Database (IMDb) for games”. Much like its cinematic counterpart, the biggest board game database not only collects all sorts of information obsessively, but also allows users to rate games on a scale from 1 (awful - defies game description) to 10 (outstanding - will always enjoy playing). These ratings are then used to rank games, with Gloomhaven occupying the top spot since December 2017.

While BGG founder Scott Alden admitted in a recent interview on the excellent Five Games For Doomsday 🗄️ podcast that he doesn’t care all that much about the rankings, gamers around the world certainly do. They would discuss heatedly any movement in the rankings, question why games X is up there while game Y is missing, and generally criticise the selection for either having too many or not enough recent releases.

Reason enough for me to take a closer look at how the rankings work and some of the maths behind it.

Generally speaking, we want to rank a game higher the better its score is. The first instinct would be to just sum up all the ratings users gave to that particular game, divide by the number of ratings, and rank games from highest to lowest. What I just described would be the arithmetic mean (or just average if you feel less fancy) of the ratings, which is simple and intuitive, but suffers from a sever defect: a game with a single rating of 10 would always sit on top of the ranking, well ahead of much beloved games with thousands of votes that couldn’t possibly be all 10s.

The easiest fix is filtering out games with less than a certain number of ratings, say 100.¹ That’s a decent enough approach, and yields the following top 5 games as of the time of writing:

Notably, those are all very recent games with relatively few ratings.² Some might consider this a feature, not a bug, but when your intention is to create a list of the best board games, you probably do want to give a nod to proven classics, and not just the latest hotness. How to balance out these ends of the spectrum is in the end a choice you have to make, and no matter what it is, People on the Internet™ will not like it.

The way both IMDb and BGG chose to tackle this issue is by essentially not trusting the ratings – at least not too much. The method boils down to assigning a new item in the database (be it movie or game) a predefined average by default, and only gradually trusting the ratings’ average as thousands and thousands of users have cast their votes. More concretely the rankings are calculated by adding a number of dummy ratings with a chosen average value, say 5.5, to each game’s regular ratings. The result is that initially each game will have a score close to 5.5, but as more users rate the game, that score will move closer and closer to the conventional mean.

BGG calls this their geek score. Mathematically speaking, it is a Bayesian average, and calculates as follows:

\[ \textrm{geek score} = \frac{\textrm{sum of ratings} + \textrm{number of dummies} \cdot \textrm{dummy value}}{\textrm{number of ratings} + \textrm{number of dummies}}, \]

where sum of ratings can be calculated either by, well, summing up all ratings or via number of ratings ⋅ average rating. Don’t worry too much about the details though – adding dummy ratings is really all you need to understand.

OK, so that’s the concept, but crucially that’s not all the details. You still need to choose how many dummy ratings you want to add and what value they should take. Since People on the Internet™ who disagree with your ranking will try to manipulate it in whatever way they can, sites are usually very cagey about said details. IMDb used to be more transparent, as was BGG, but now we have to dig a little deeper.

Let’s start from the easier of the two, the value of the dummy ratings. It is commonly chosen to represent some prior mean, i.e., some decent estimate of the rating a new game in the database would have. A frequent choice would be to use the average rating across all games. It’s a fair assumption – without further information about a game, we don’t know if it’s any better or worse than the average game. However, Scott Alden actually gave away the answer in that interview from the beginning: BGG chose the dummy value to be 5.5. Their rationale is that ratings range from 1 through 10, so 5.5 is the midpoint. Of course, people tend to rather play and rate much more the games they like, and so the average rating is around 7. Opting for the lower value here is part of the design of the ranking: it means a new game would enter the ranking rather at the end of the pack. On the other hand, using the mean as the dummy value means a new game is placed more or less in the middle. It is worth mentioning that IMDb does use the mean (or at least used to), but they only ever publish the top 250 movies, and don’t care about the crowd behind.

The other value, the number of dummy ratings, requires more work. Because some of the details and data are unknown, we cannot actually pin down the exact number that BGG is using. Instead, we’ll try three different approaches, and compare their results.

Formula

On the surface, this should be super easy to solve: in the formula above, we know every single value but the number of dummy ratings. BGG publishes the number of ratings, their arithmetic mean, and the “geek score” or Bayesian average for every game, and we know that the dummy value is 5.5. With a little high school algebra we solve the formula for number of dummies:

\[ \textrm{number of dummies} = \textrm{number of ratings} \cdot \frac{\textrm{average rating} - \textrm{geek score}}{\textrm{geek score} - \textrm{dummy value}} \]

Now we should be able to plug in those values for any given game, say Flamme Rouge, and get the result. With 10,936 ratings that average 7.562, and a geek score of 7.266, this yield:

\[ \textrm{number of dummies} = 10936 \cdot \frac{7.562 - 7.266}{7.266 - 5.5} \approx 1830. \]

So, there’s about 1830 dummy ratings, end of story. Right? Unfortunately, not quite. When computing this formula for different games, the results vary wildly, as you can see from this histogram over the results for the same calculation with other games:

And this plot is even cropped, the results vary from -1.4 million to +810 thousand, though some 90% lie within the above range, with a mean of around 1604 and a median of around 1590.

What’s going on, why are the results so inconsistent? The problem is the ranking’s secret sauce. Both IMDb and BGG stress is that they only consider regular voters for their rankings. That’s the most mysterious part of the system as it’s the easiest to manipulate, so we’ll just have to take their word for it. For this investigation it means that the average rating BGG publishes includes all the ratings, but the geek score might not.

Still, clearly something is happening around the 1600 ratings mark, so we are at least getting closer to an answer. If exact calculations won’t work, maybe we can approximate the correct value instead?

Trial & error

Let’s take a step back here. What we’re really trying to achieve here is not finding the exact formula for that mysterious “geek score”, but rather recreate the BGG ranking. That is, we want to find the values in the above formula, such that the resulting ranking matches BGG’s ranking as closely as possible. Luckily, statistics has all the tools we need. Spearman correlation measures rank correlation – just what we need. This will be 1 if both rankings sort in exactly the same way, 0 if there’s no relation, and -1 if they sort exactly the opposite way. Again, don’t worry about the details, just trust the maths.

What we can do now is fairly simply and quickly compute the rankings for different number of dummy ratings, and pick the value with the highest Spearman correlation. Without further ado, here are the results:

The best correlation of around 0.996 is achieved with 1488 dummy ratings. However, it is worth noticing that the changes in the correlation are very, very small throughout the range we examined here (1000 to 2500), so let’s dig still a little deeper.

Optimisation

What we have here at hand is actually a classic optimisation task: a real valued function in one unknown (or two if we allow a variable dummy value as well) which we’d like to maximise. This is a well-studied field, with many fast and simple implementations that provide us the solution in no time. Unsuprisingly, we get the same result as above: the best possible correlation is 0.996 with around 1488 dummy ratings.

But since we made it this far, let’s take it one step further. So far, we tried to optimise the correlation in order to recreate BGG’s ranking. However, we can also try to recreate the actual geek scores. That is, we can look for the number of dummy ratings that will yield the closest to the actual geek score with our calculations. What exactly we mean by “closest” is up to us to define. A common metric is the mean squared error.³ It’s not worth getting into the maths here either, but the general idea is that we want to punish outliers in our estimates more (qudratically so) the further away they lie from the actual datapoint. Long story short, this yields a minimum for around 1636 dummy ratings.

Let’s take one last swing and see what happens if we don’t fix the dummy value at 5.5 but allow that to be variable as well. This is no problem for the optimisation algorithm and yields the following results:

the best correlation with 1942 dummy ratings of 5.554, and
the least squared error with 1616 dummy ratings of 5.494.

Either of those improvements in the performance metrics are hardly noticable (in fact insible after rounding), but they do confirm nicely a dummy value of 5.5.

Conclusion

All things consider, we can be confident that BoardGameGeek calculates their rankings by adding around 1500 to 1600 dummy ratings of 5.5 to the regular users’ ratings. What exactly constitutes a regular user, and what ratings might be discarded due to shilling, remains a well guarded secret though. Note that the number of dummies is pegged to the overall number of ratings, so this is a moving target, and the calculations would change as time passes.

Now I must applaud anybody who actually made it all the way through this pretty dry and technical article. The real reason why I dwelled so much on the ratings, and how they are compressed into the BGG rankings, is to get a feeling of what’s going on behind the scenes, what the can express, and what they cannot or even do not try to express. Another major take-away is that any of these decisions are choices that need to be made and that come with certain tradeoffs – like them or not.

In this particular case, I have the feeling that both the Cult of the New™ and connoisseurs of classic games are equally unhappy about the BGG top 100, which one should probably consider a compliment.

Alternative rankings

I’ll send you off with some rankings that were obtained by making different choices for the two values that we discussed throughout this article: the number of dummy ratings and their value.

Using the ratings average as dummy value

I’ve mentioned before that the average rating across all games is around 7 – a little¹ more precisely 7.08278. What if we chose that as the dummy rating, but left their number at 1600? The result should be a ranking that is a little friendlier to newer titles with fewer ratings as their score isn’t dragged all the way down to 5.5 in the beginning.

Sure enough, the brand new Jaws of the Lion with less than 3000 ratings already shows up in the top 10. The other game that sticks out here is Kingdom Death: Monster. This Kickstarter success story clearly attracted a lot of enthusiasts, but not necessarily the mass.

Using the top 250 number of ratings

Just like IMDb publishes only their top 250 movies, we can consider the same and crank up the number of dummy ratings. A good number seems to be the 250th most rated game on BGG, which has been rated 12,014 times. Using BGG’s standard dummy value of 5.5, we obtain a ranking that is much more skewed towards proven classics:

The most recent release on this list is Gloomhaven, but we also meet again old BGG #1’s: Puerto Rico and Agricola.

Combining both!

Finally, let’s do what IMDb does (or used to do), and add to each game’s ratings 12,014 dummy ratings of 7.08278:

The effects of more, but higher dummy ratings seem to almost cancel each other out. Compared to BGG’s actual top 10, only Twilight Imperium and Gaia Project are missing, otherwise this ranking looks very familiar. Turns out, BGG did a pretty good job designing its ranking!

Read the second part here where we nail down the number of dummy ratings to the dot.

PS: You can find the notebook I used to do all the calculations on Kaggle.

PPS: Turns out that GoDataDriven did almost the same calculation 🗄️ three years ago – even with the same title! Back then, they estimated that BGG added 725 dummy ratings. Jorge Nieva replicated their analysis with a more recent dataset, which thankfully yielded a very close match to our result: 1594 dummy ratings.

Throughout this article I only considered games with at least 100 ratings, mostly to ensure that the very long tail of games with few ratings won’t unduely skew the results. However, most of the calculations would only change in some negligible decimals when including all games. ↩︎ ↩︎
Jaws of the Lion is something of an exception here and will undoubtably shoot into the BGG top 10 very soon. In fact, it might be the only game with the potential to unseat Gloomhaven as the number 1. ↩︎
It’s probably even more common to use the root mean squared error, but for boring mathematical reasons, it doesn’t make a difference when it comes to optimisation. In fact, we could even drop the word mean from our metric and still obtain the same optimal point, so let’s not dwell on this. ↩︎