Statistics on Analysis Paralysis

Teaching Elo to Play with Friends

contact@recommend.games (Recommend.Games) — Fri, 12 Dec 2025 12:00:00 +0200

At some point this year, I let my laptop run flat-out for almost two weeks just to answer one question: how much of a four-player board game is “skill” and how much is “luck”? That sounds excessive, but there was a catch: before I could even start those simulations, I had to fix a basic problem. Elo – the rating system we’ve been happily using so far – only really knows how to handle one-on-one duels.

This article is the missing technical chapter in the series. In part 1 we met Elo and learned how it turns match results into skill ratings. In part 2 we sent those ratings to the Crucible to predict the next World Snooker Champion. And in part 3 we stole a clever idea from Dürsch, Lambrecht and Oechssler to turn the spread of Elo ratings in a two-player “toy universe” into a kind of skill-o-meter: a way to say whether a game behaves more like a 30%-skill world or an 80%-skill world.

There’s one obvious gap left: most modern board games aren’t tidy head-to-head affairs. Around a real table you’ll usually find three, four, sometimes five players battling it out in CATAN, Brass, Gaia Project or whatever your current obsession is. If we want to use our shiny skill-o-meter on those games, we first have to teach Elo how to cope with real multiplayer tables instead of just faking them as a stack of two-player matches.

Fair warning: this part is even more technical than part 3. We’ll talk about probability matrices, permutations and a scary-looking formula or two. If that’s not your thing, you’re still very welcome to skim the maths-heavy bits – I’ll keep pointing out the important intuitions along the way. The payoff is worth it: by the end of this article, we’ll have a principled multiplayer Elo system and a checked-and-calibrated skill-o-meter that still works when three, four or five people sit down to play.

Why two-player Elo isn’t enough for modern games

Elo’s original paper was targeted at chess, so naturally it was only concerned with two-player games. Likewise, everything I’ve talked about in this series so far has assumed a simple head-to-head match: one player vs another, winner takes the Elo chips.

If we want to apply our shiny “skill-o-meter” from part 3 to the games we actually play, we need to teach Elo how to handle true multiplayer tables instead of just faking them as a bunch of two-player matches.

How people fake multiplayer Elo (and why it’s not quite right)

If you’re like me and spend a slightly embarrassing amount of your free time on Board Game Arena, you might have noticed their Elo implementation. They simply treat multiplayer games as a collection of 1‑vs‑1 battles. So if Alice, Bob and Carol play a game, their Elo calculations treat this as three matches: Alice vs Bob, Alice vs Carol and Bob vs Carol. If Alice indeed won the game, Bob came in second and Carol last, Alice would win both her “virtual” matches and Bob his against Carol. Elo ratings would then be updated according to the regular formula, with $K$ “adjusted for player count” (I didn’t find an up-to-date source as to the details).

Conceptually, this is a neat hack but not quite right: it pretends Alice actually played two independent duels against Bob and Carol, even though in reality all three interacted in the same shared game state and their decisions affected each other at the same time.

Note that for an $n$-player game there are ${n \choose 2} = \frac{n(n-1)}{2}$ pairings, so the number of updates grows quadratically with player count. This kind of growing complexity can really come back to bite you in the behind when it comes to compute, but (a) luckily we don’t need to worry about matches with hundreds of players in tabletop gaming and (b) it could be much worse, as we shall see in a minute…

A more principled multiplayer Elo: ranking probabilities

In part 3, we already leaned on a neat idea by Peter Dürsch, Marco Lambrecht and Jörg Oechssler from their paper “Measuring skill and chance in games” (2020). There we used their framework to turn the spread of Elo ratings into a “skill-o-meter” for two-player games. In this article, we’re going back to the same well: DLO also propose a way to run Elo on proper multiplayer tables, and that’s exactly the tool we need for modern board games.

From table results to expected payoffs

Dürsch et al suggest a more principled way to deal with multiplayer tables. Let $n$ be the number of players in the match. Instead of pretending everyone played everyone else in separate duels, they directly model the whole finishing order at once.

The first ingredient is an $n\times n$ matrix of probabilities:

\[ p_{ij} = P(\text{player $i$ finishes in position $j$}). \]

You can read row $i$ as “what’s the chance player $i$ finishes 1st, 2nd, …, last?” and column $j$ as “who is most likely to end up in position $j$?”.

Just like in the two-player case, we need a numerical payoff to compare expectations with reality. For an $n$-player game we give the winner $n-1$ points, the runner-up $n-2$, all the way down to 0 for last place.¹ If there are ties, we give each tied player the average of the payoffs they straddle. That gives us the expected payoff for player $i$:

\[ e_i = E[\text{payoff for player $i$}] = \sum_{j=0}^{n-1} p_{ij} (n - 1 - j). \]

Once we have that, the Elo update looks exactly like before. Let $a_i$ be the actual payoff (from the final ranking, scaled in the same way). We compare $a_i$ to $e_i$, and shift the rating in the direction of the surprise:

\[ r_i \leftarrow r_i + \frac{K}{n-1} (a_i - e_i). \]

The factor $1/(n-1)$ just normalises things so that one whole game still corresponds to about $K$ “chips” moving around, as in the two-player version.

From Elo ratings to ranking probabilities

This is where things get a bit heavy. If you’re mostly here for the big picture, feel free to skim or even skip the formulae in this section — I’ll summarise the important part again at the end.

Conceptually, what we want is simple: for a given set of Elo ratings, we assign a probability to each possible finishing order of the players. Stronger players should be more likely to end up near the top, weaker ones near the bottom. Once we have those probabilities, we can add them up to get the chance that a particular player finishes in a particular position.

Formally, we write a possible ranking as a permutation $\tau$ of ${0, \dots, n-1}$, where $\tau(j)$ tells us which player ends up in position $j$ (with $j=0$ for the winner, $j=1$ for second place, and so on). The probability of seeing a particular ranking $\tau$ can be written using the chain rule of probability:

\[ P(\tau) = P(\text{players $\tau(0), \dots, \tau(n - 1)$ on positions $0, \dots, n - 1$}) \\ = \prod_{j=0}^{n-1} P(\text{player $\tau(j)$ on position $j$} \mid \text{players $\tau(0), \dots, \tau(j - 1)$ fixed above}). \]

To estimate those conditional probabilities, Dürsch et al use the softmax over Elo ratings. Softmax is just the multiplayer cousin of the Elo win-probability formula: you take a “strength score” for each player, exponentiate it, and then divide by the sum so that everything adds up to 1. At each step $j$, we look at the players who haven’t been placed yet and assign probabilities proportional to $10^{r / 400}$, just like in the two-player Elo formula. If we write $r_i$ for the current rating of player $i$, this gives:

\[ P(\text{player $\tau(j)$ on position $j$} \mid \text{players $\tau(0), \dots, \tau(j - 1)$ fixed above}) \\ = \frac{10^{r_{\tau(j)} / 400}}{\sum_{k=j}^{n-1} 10^{r_{\tau(k)} / 400}}. \]

Plugging this into the chain rule expression yields a compact formula for the probability of a full ranking $\tau$:

\[ P(\tau) = \prod_{j=0}^{n-1} \frac{10^{r_{\tau(j)} / 400}}{\sum_{k=j}^{n-1} 10^{r_{\tau(k)} / 400}}. \]

Now, to get the entries of our probability matrix, we just have to sum over all rankings that put a given player in a given position. Remember that $p_{ij}$ is the probability that player $i$ finishes in position $j$. With the convention $\tau(j) = i$ meaning “player $i$ sits in position $j$”, we have:

\[ p_{ij} = \sum_{\tau \text{ with } \tau(j) = i} P(\tau). \]

If the formulae lost you at some point, that’s OK — the story is simple: we assign a probability to each possible finishing order based on the Elo ratings, and then sum those probabilities to find out how likely each player is to end up in each position. That’s all you really need to remember from this section.

Does this really generalise two-player Elo?

You might still wonder if it’s really justified to call this a generalisation of two-player Elo, since it looks rather different at first glance. The crucial sanity check is that when we only have $n = 2$ players at the table, all of this machinery collapses back to the usual head-to-head model: there are only two possible rankings, the probability matrix reduces to the familiar win–loss probabilities, the payoff vector $(1, 0)$ just scores win vs loss, and the update rule becomes exactly the original Elo formula again.² You don’t need to wade through the algebra – the important point is that for ordinary two-player encounters, this system behaves just like classic Elo.

The price of doing it properly: combinatorics and compute

There is one big catch we’ve glossed over so far. To calculate the entries of the probability matrix $p_{ij}$, we have to sum $P(\tau)$ over all possible rankings $\tau$. If you remember your combinatorics basics, you’ll know that there are $n!$ permutations of $n$ players – a function that grows even faster than exponential. In other words: a straightforward implementation of this model is computationally very expensive.

Does this mean the whole approach is doomed? Luckily, not quite. Most board games have at most five or six players, and $6! = 720$ is big but still perfectly manageable on a modern computer. That covers the vast majority of situations we care about in tabletop gaming.

For higher player counts there are more efficient tricks (for example dynamic programming and Monte Carlo approximations) that avoid looping over all permutations explicitly. I’m not going to go into the details here; if you’re curious, you can have a look at the implementation in the code for this article – but for our purposes it’s enough to know that the full model is tractable for realistic games.

Multiplayer p-deterministic games

Right, after so much theory you deserve something a bit more concrete. Real-world applications will come in the next article; for now, there’s still one more thing to check: do the multiplayer versions of the $p$-deterministic game behave in the same way as the two-player toy world we built in part 3?

The setup remains almost the same. We fix an underlying skill ranking for all players. For each game, we flip a weighted coin: with probability $p$ we play a game of pure skill, where players finish in order of their underlying strength; with probability $1-p$ we play a game of pure chance, where the finishing order is just a random permutation of the players. It’s the same toy universe as before, just with more than two players sitting at the table each time.

The σ vs p benchmark still holds for up to 15 players

With this multiplayer version of the $p$-deterministic game in hand, we can run the same kind of simulations as before. For each choice of $p$ and for player counts between 2 and 15, we let lots of games play out, calibrate $K$ on the simulated match data, compute the resulting Elo ratings and record their standard deviation $\sigma$. Plotting $\sigma$ against $p$ for each player count gives us this family of curves:

All of these curves are smooth and strictly increasing: as we turn up $p$ and let skill matter more often, the Elo spread $\sigma$ grows, just like in the two-player case. More interestingly, when we plot these player counts from 2 up to 15, the points for different player counts are essentially indistinguishable: for each value of $p$, all the coloured dots sit almost exactly on top of each other. Any tiny visible wobble at very high $p$ is well within the limits of simulation noise and numerical quirks.

That’s precisely the behaviour we were hoping to see. Empirically, in this toy universe $\sigma$ is effectively a function of $p$ alone and — within our numerical precision — invariant to how many players sit at the table, even up to 15. In practical terms, this means that if we measure a standard deviation $\sigma$ in a real three-, four- or five-player game, we can safely read off a corresponding “$p$-skill world” from this benchmark without worrying about the exact player count.

Talking of the computational effort: getting this last plot alone down to only about two weeks of wall-clock time on my poor laptop took a fair bit of optimisation. The result might look a little underwhelming after all that build-up, but that’s exactly the point: after grinding through all those simulations, the curves stubbornly agree that player count basically doesn’t matter. 🔥😅🤓

Where this leaves us (and what’s next)

We’ve covered a lot of ground in this article, but the payoff is twofold.

First, we now have a principled way to run Elo on real multiplayer tables. Instead of faking CATAN or Brass as a pile of head-to-head duels, we can model the whole finishing order at once, get sensible expected payoffs for each seat, and update ratings in a way that reduces to classic two-player Elo when there are only two people at the table.

Second, we’ve stress-tested our “Elo-as-a-skill-o-meter” from part 3 in a richer toy universe. In those $p$-deterministic worlds, the standard deviation $\sigma$ of Elo ratings turns out to depend almost entirely on $p$ and, within numerical accuracy, not on how many players sit down to play. That means $\sigma$ really does behave like a calibrated skill dial we can use for 2–6 player games.

Put together, this gives us the toolset we wanted: given real multiplayer game logs, we can (a) fit Elo using the multiplayer update, (b) calibrate $K$ on predictive accuracy, (c) read off the resulting $\sigma$ and map it to a “skill fraction” $p$ using our benchmark curve.

Next time, we’ll finally unleash this machinery on actual board games. We’ll look at real play logs, see which games behave more like 30%-skill worlds and which ones look closer to 80% skill, and maybe settle a few pub arguments along the way. 🤓

Dürsch et al use a flexible payoff structure which makes the formulae and implementation more confusing. For our purposes, the fixed payoff based on ranks is enough, so I tried to keep things simple. ↩︎
If you’re itching to do the algebra yourself, be my guest — that’s the unofficial “exercise to the reader” for this section. I decided you didn’t need to watch me juggle minus signs for a page. ↩︎

Elo as a Skill-O-Meter

contact@recommend.games (Recommend.Games) — Fri, 05 Dec 2025 12:00:00 +0200

Whether a game counts as “skill” or “chance” isn’t just a pub argument — in many countries it’s a legal distinction. Roulette and blackjack live on the “chance” side; tennis and chess are filed under “skill”. Different rules, different taxes, different ways for people to lose money.

The trouble is that this line is usually drawn by tradition and gut feeling. Is poker really “more skill” than backgammon? Is snooker closer to roulette or closer to chess? A group of economists tried to answer that question more systematically: instead of arguing, measure how “skill-heavy” a game is in practice by looking at the Elo ratings of all its players. We’ll meet their work properly in a bit.

In this article I want to steal that idea for board games. So far we’ve used Elo to track individual player strength; this time we’ll go one level up. Instead of asking who is strong, we’ll look at the whole distribution of Elo ratings in a game and see what its spread can tell us about luck and skill — turning Elo into a kind of “skill-o-meter”.

From Elo ratings to skill distributions

By now the basics of Elo should be familiar: each player gets a rating that reflects their playing strength, rating differences go into a simple formula to give expected win probabilities, and after each match we update those ratings based on whether players beat expectations. If you want the full story (including all the maths and the logistic regression detour), part 1 has you covered; here we’ll treat Elo as a black box that turns match results into reasonable estimates of player skill.

In part 2 we applied this system to predict the 2025 World Snooker Champion. The model’s favourite, John Higgins, didn’t manage to win his fifth title, but it did give eventual winner Zhao Xintong a 10.6% chance when the bookies only gave him 5.9%. I’ll take that as a personal win — and as evidence that Elo isn’t just numerology, it really does capture something about players’ skills.

Wider distributions, more skill

What we’re really after is a way to say how much skill is involved in a game, and to compare different games with each other. The individual Elo numbers are just a stepping stone. To get anywhere, we have to zoom out and look at the whole distribution of skills, as measured by all players’ Elo ratings.

The basic idea is simple: if a game is mostly luck and players’ decisions don’t matter much, nobody can reliably stay ahead for long. You’ll see some winning streaks, but they’ll wash out again, and everyone’s ratings will cluster around 0. In a game where skill really matters, the strongest players win more often than they lose and slowly drift away from the pack. The result is a much wider Elo distribution: a long tail of very strong players and a long tail of weaker ones.

A first look: snooker vs tennis

Let’s make this more concrete. We’ve already calculated Elo ratings for snooker, so let’s compare it to another English upper-class sport played on a green surface: tennis. Can you tell which one is more skill-based? To get a more objective answer, we look at the Elo distributions for both and see which one is wider:

According to this plot, the Elo ratings of snooker players have a much higher peak and shorter tails, which suggests that outcomes are more influenced by luck. Tennis — at least on the WTA, the women’s tour¹ — seems to show a wider spread and therefore more room for skill. But how do we know these distributions are even comparable? And can we turn that vague “more luck, more skill” into an actual number?

Turning spread into a skill measure

To answer these questions, we need to properly dive into the science. 🧑‍🔬

The one neat trick economists use to measure skill (that gamers can steal)

I’m going to lean on a neat idea by Peter Dürsch, Marco Lambrecht and Jörg Oechssler, from their paper “Measuring skill and chance in games” (2020). They come from an economics background and originally cared about gambling regulation, but the trick itself is much more general: take the Elo ratings for all players in a game, look at their distribution, and from that pin down a single number that tells you where the game sits on the spectrum between “pure chance” and “pure skill”. That’s exactly what we’re trying to do here — just for board games instead of casinos. 🤑

So we’ll follow their lead and focus on the spread of the Elo distribution as our measure of how much skill shows up in a game.

Standard deviation of Elo ratings

The mathematical measure for the spread of a distribution is its standard deviation $\sigma$. The wider the distribution, the larger its standard deviation. Roughly speaking, it’s the expected (squared) difference from the mean. In our setting, that means $\sigma$ tells us how far, on average, players’ skills lie from the “average” player: a bigger $\sigma$ means the field is more spread out, with larger typical gaps between players — exactly the sort of quantity we want to look at.

So from now on, whenever I talk about the “amount of skill” we see in a game, I’ll use the standard deviation of its Elo ratings, $\sigma$, as the proxy.

The problem with K

There’s an important caveat: Elo ratings and their distribution crucially depend on $K$, the update factor. If you remember the metaphor from the first article, $K$ is the number of “skill chips” the players put into the pot each game. Higher stakes lead to a wider spread; if almost no chips change hands, everyone ends up with roughly the same number and the spread stays tiny.

A natural idea would be to just fix one $K$ for all games. Unfortunately, that doesn’t work either. Imagine two ladders for exactly the same game with the same population of players. In ladder A, everyone plays a handful of games per year; in ladder B, the same people grind hundreds of games a month. We now run Elo with the same $K$ on both datasets. In ladder A only a few “skill chips” ever change hands before the season is over, ratings barely have time to drift apart, and the final distribution stays fairly tight. In ladder B, chips slosh back and forth for thousands of rounds; random streaks get amplified, the system has time to separate the strong from the weak, and the rating spread ends up much wider. The underlying game and the underlying skills are identical, yet the dispersion of Elo ratings depends heavily on how often people play and how long we observe them. Fixing $K$ globally doesn’t make the spread an intrinsic property of the game — it just bakes in arbitrary design decisions about volume and time.

Calibrating K from the data

What Dürsch et al suggest instead is to calibrate $K$ from the data, so that the Elo ratings are as good as possible at the job they were designed for: predicting who wins. Remember how Elo works: we take the pre-match ratings $r_A$ and $r_B$ of two players A and B, and feed the difference into a logistic formula to get the predicted win probability for A:

\[ p_A = \frac{1}{1 + 10^{-(r_A - r_B) / 400}}. \]

After the match, we compare this prediction with $s_A$, the actual outcome of the match ($s_A = 1$ if A won, $s_A = 0$ if they lost and $s_A = 0.5$ in case of a tie), and update:

\[ r_A \leftarrow r_A + K (s_A - p_A). \]

The basic assumption in DLO’s approach is that $K$ is “optimal” if these prediction errors $s_A - p_A$ are, on average, as small as possible. For a given $K$ and a set of matches $t \in {1, …, T}$, we look at the squared errors and minimise their mean:

\[ K^* = \argmin_K \frac{1}{T} \sum_{t=1}^T (s^{(t)} - p^{(t)})^2. \]

This mean squared error is a standard loss for training machine learning models; when we apply it to probabilistic predictions like this, it’s known as the Brier loss. We can search over $K$ to find $K^*$, the update factor that makes Elo’s predictions as accurate as possible on that dataset.² Different games (and different datasets) will generally end up with different $K^*$. Once we’ve found $K^*$, we can run Elo with that value and then look at the resulting rating distributions.

Why K* is not our skill metric

Before we happily run Elo with $K^*$ and stare at the resulting distributions, let’s pause and ask what $K^*$ itself is telling us. Earlier I compared $K$ to a step size in an iterative learning process: a larger $K$ means we take bigger steps on each update and let a single match pull the ratings around more. If the “optimal” $K^*$ is large, doesn’t that mean each game is very informative about the players’ skills? That sounds suspiciously close to what we’re trying to measure. Can we just use $K^*$ as our coveted luck–skill number?

Not quite. First of all, as we’ve already discussed above, the optimal $K$ depends strongly on the player population. Larger sets of matches will tend to have smaller $K^*$, even if the underlying skill levels are exactly the same, simply because with more data you don’t need to react as violently to each individual result. That’s why we have to calibrate $K^*$ on the exact dataset we’re using.

Second, two games might demand the same underlying skills, but still have very different learning curves: some are slow and steady, others click after a single “epiphany”. Those learning dynamics also feed into $K^*$: in a game where people improve in big jumps, you’ll see a different “optimal” step size than in a game where everyone creeps up gradually. So even if two games are equally skill-based in the end, their $K^*$ values can be quite different, and comparing them would be misleading.

Luckily, the standard deviation of the Elo distribution is much more robust to those issues than $K^*$ itself: it mostly cares about where everyone ends up, not about how fast they got there.

We now have the theoretical foundation to compute Elo distributions and their standard deviation. What we still need is actual game data. I’ve already teased how this applies to snooker and tennis, and in a later article we’ll look at many more concrete examples.

A toy universe of luck and skill

Before we get there, though, I want to take a closer look at a synthetic example. There are two good reasons for this extra step. First, it gives us a simple little sandbox where we can see what’s going on and sanity-check that the method behaves as we expect. Second, it lets us build an excellent benchmark that will help us interpret those fairly abstract standard deviations later on.

Extreme worlds: pure chance vs pure skill

Let’s start with two extreme scenarios. First, a game of pure chance, where the winner is literally decided by a coin toss. Second, a game of pure skill, where there is some fixed underlying skill ranking and the stronger player always beats the weaker one. What would the Elo distributions look like in those two imagined worlds?

In the totally random case, no player ever has any real advantage over another, so the “skill chips” just get tossed back and forth. Some winning streaks will occur, of course, but in the long run they’re balanced by losing streaks. Elo will keep nudging ratings back towards the middle, and everyone’s rating will hover near 0. The overall spread $\sigma$ settles into a very narrow band around 0.

In the opposite extreme, there is a fixed skill ranking, and the strongest player always beats everyone else. This top player will keep siphoning rating points from their opponents and never really settle at a final value. Elo is designed so that very large skill differences lead to only tiny rating changes, but in a world of perfect skill, there is always at least one opponent – the second-best player – who still gives them a little positive update every time they meet. The second-best player in turn keeps gaining points from everyone below them, and so on down the ladder. As a result, the strongest players drift further and further away from the pack, while the weakest ones sink lower and lower. In principle, the spread of ratings can grow without bound.

The p-deterministic game

With the extremes out of the way, we can now blend them into an intermediate case: the $p$-deterministic game. The idea is simple. We fix an underlying skill ranking for all players. Before each match, we flip a weighted coin: with probability $p \in [0,1]$ we play a game of pure skill, where that ranking decides the winner; with probability $1-p$ we play a game of pure chance, where the winner is chosen at random. This little Gedankenspiel is easy to understand and reason about. It gives us an idealised example of a game with “roughly $p$ parts skill and $1-p$ parts luck”, and it serves as the benchmark I promised — something we can later compare real games against. And because the rules are so simple, we can easily run simulations and calculate the resulting Elo distributions:

What simulations tell us

The first plot already shows the basic pattern: as we turn up $p$ and let skill matter more often, the Elo distribution gets wider and wider. To make this easier to see, we can just take the standard deviation $\sigma$ of each distribution and plot it against $p$:

The result is a smooth, monotone curve: higher $p$ consistently leads to a larger Elo spread $\sigma$. That gives us exactly what we wanted — a way to translate those abstract standard deviations into a more tangible “skill fraction” $p$. Later on, when we look at real games, we’ll be able to say “this game behaves roughly like a 70%-skill world” by matching its Elo spread to this benchmark curve.

What’s next

There’s still one big limitation left: everything so far has assumed two-player games. In the next part of this series, we’ll teach Elo to handle real multiplayer tables — the kind we actually have in modern board games — and only then move on to real-world data.

Interestingly, the Elo distribution for men’s tennis (ATP) looks more similar to the one for snooker than women’s tennis. ↩︎
Remember that $K=42$ I used in the snooker article? I promised I’ll explain in excruciating depth where it came from and I think I kept my promise. ↩︎

Game Designer Hall of Fame

contact@recommend.games (Recommend.Games) — Fri, 17 May 2024 12:00:00 +0300

Spiel des Jahres 2024 nominations are just around the corner (watch this space for our annual predictions), but I wanted to take this opportunity for a look back over the history of the award. Specifically, I wanted to see which designers have been most successful at the Spiel, Kennerspiel and Kinderspiel des Jahres awards.

The first red meeple was given out in 1979. Since 1989, the jury also awarded the Kinderspiel des Jahres for children’s games, initially as an annual special award, and from 2001 as its own award with the blue meeple. (In what follows I count those special awards as regular Kinderspiel des Jahres winners.) Finally, in 2011, the Kennerspiel des Jahres was introduced for (slightly) more complex games with an anthracite meeple, again following a couple of special awards with a clearly similar intent, but less steady, so they just stand as special awards on their own in the statistics.

Games

Overall, the jury included 831 games on its various longlist across the three categories over its 45 year history, with (naturally) 45 Spiel, 35 Kinderspiel and 13 Kennerspiel des Jahres winners.¹ This is the overall number of games:

Award	Winner	Nominated	Recommended	Special award
Spiel	45	109	482	11
Kennerspiel	13	39	76	2
Kinderspiel	35	95	260	1

A quick word about the data: As usual, I rely on BoardGameGeek (BGG) for data about the games and their designers. If a game is not in the database or if the designer entry is wrong, the statistics here will necessarily be wrong too. I’ve collected data about the different Spiel des Jahres awards manually over the years, so that’s another potential source of errors. If you’re reading this from the future, I should point out that the statistics here should be complete up to and including Spiel des Jahres 2023.

The current format of having a longlist (which become the recommendations), out of which three games are selected for the shortlist (which are the nominated games), out of which one game will be given the award, has been in place since 2011 with the introduction of the Kennerspiel des Jahres. Formats have varied throughout the decades and I’ve tried to map them to the current situation as much as possible.

I’ve already mentioned that the first 12 Kinderspiel des Jahres winners were really special awards. There was another special award given during the first 18 years between 1979 and 1997 for “beautiful games”. Since this article is meant to focus on designers, not artists, it felt like those games clutter the statistics somewhat, so I treated them as recommendations rather than special award winners.

So without further ado, let’s take a look at the designers.

Designers

Over the course of 45 sessions, the jury has mentioned games by 566 different designers. By far the most successful designer is Wolfgang Kramer who has won Spiel des Jahres five times (four of those were co-designs) and Kinderspiel des Jahres once between 1986 and 2000. During that time period, Klaus Teuber won Spiel des Jahres four times, so during those 15 years Wolfgang & Klaus shared 11 awards between them. The record for most games on the longlists is held by Reiner Knizia with 36 games, which makes it even more surprising that he has “only” won two awards: one each of Spiel des Jahres and Kinderspiel des Jahres – both in 2008 (and both largely forgotten today, unlike his other timeless designs).

Overall, 94 designers have won at least one award (111 if we include special awards), with 21 of them winning more than one (22 including special awards). Only 8 of those have won their awards in more than one category: Wolfgang Kramer, Inka & Markus Brand, Alex Randolph, Reiner Knizia, Bruno Cathala, Antoine Bauza and Steffen Bogen. To this date, no designer has won in all three categories.

Looking at the shortlists (including winners), the jury nominated 217 different designers over the years. Of those, only 22 have been nominated in more than one category and just 4 designers can claim the distinction of being nominated in all three categories: Wolfgang Kramer, Antoine Bauza, Rüdiger Dorn and Wolfgang Warsch.

As mentioned earlier, the jury has recommended games by 566 different designers, with 68 of them appearing in more than one category and 9 in all three. In addition to the 4 mentioned above, those are Inka & Markus Brand, Reiner Knizia, Bruno Cathala and Roberto Fraga.

I’ll leave you with some more detailed statistics below. I hope you enjoy those and they help you bridge the time until the nominations are announced on June 11th. 🤩

Designer records

Overall

Most games on the longlist: 36
- Reiner Knizia
Most games on the shortlist: 13
- Wolfgang Kramer
- Reiner Knizia
Most wins: 6
- Wolfgang Kramer
Most games on the shortlist without win: 5
- Jürgen P. Grunau

Spiel des Jahres

Most games on the longlist: 25
- Reiner Knizia
Most games on the shortlist: 9
- Wolfgang Kramer
Most wins: 5
- Wolfgang Kramer
Most games on the shortlist without win: 5
- Rüdiger Dorn

Kennerspiel des Jahres

Most games on the longlist: 4
Most games on the shortlist: 3
- Stefan Feld
Most wins: 2
Most games on the shortlist without win: 3
- Stefan Feld

Kinderspiel des Jahres

Most games on the longlist: 13
- Kai Haferkamp
Most games on the shortlist: 5
- Reiner Knizia
- Heinz Meister
Most wins (incl special award): 3
- Marie Fort
- Wilfried Fort
Most wins (main award): 2
Most games on the shortlist without win: 3
- Jürgen P. Grunau

Designers in different categories

Longlist (incl shortlist and winners)

9 designers appear in all 3 categories
68 designers appear in at least 2 categories
566 designers appear in at least 1 category

Shortlist (incl winners)

4 designers appear in all 3 categories
22 designers appear in at least 2 categories
217 designers appear in at least 1 category

Winners

0 designers appear in all 3 categories
8 designers appear in at least 2 categories
94 designers appear in at least 1 category (111 incl special awards)

Winners per category

Spiel des Jahres

342 designers had a game on the longlist
105 designers had a game on the shortlist
58 designers won the award (incl special awards)
48 designers won the main award

Kennerspiel des Jahres

95 designers had a game on the longlist
52 designers had a game on the shortlist
20 designers won the award (incl special awards)
14 designers won the main award

Kinderspiel des Jahres

206 designers had a game on the longlist
86 designers had a game on the shortlist
41 designers won the award (incl special awards)
40 designers won the main award

Spiel & Kennerspiel des Jahres

22 designers had a game on both longlists
9 designers had a game on both shortlists
1 designer won both main awards

Spiel & Kinderspiel des Jahres

55 designers had a game on both longlists
15 designers had a game on both shortlists
5 designers won both main awards

Kennerspiel & Kinderspiel des Jahres

9 designers had a game on both longlists
6 designers had a game on both shortlists
2 designers won both main awards

Spiel, Kennerspiel & Kinderspiel des Jahres

9 designers had a game on all longlists
4 designers had a game on all shortlists
0 designers won all main awards

Designer Hall of Fame

This table includes designers with at least 2 wins or 5 nominations across all categories. Each entry shows the number of wins / nominations / recommendations for that designer in each category. Special awards are added in parentheses.

Designer	Spiel	Kennerspiel	Kinderspiel	Total
Wolfgang Kramer	5 / 4 / 12	0 / 1 / 0	1 / 2 / 3	28
Klaus Teuber	4 / 0 / 3	0 / 0 / 0	0 / 1 / 1	9
Michael Kiesling	3 / 3 / 4	0 / 2 / 0	0 / 0 / 0	12
Inka Brand	0 / 0 / 2	2 / 0 / 0	1 / 2 / 0	7
Markus Brand	0 / 0 / 2	2 / 0 / 0	1 / 2 / 0	7
Alex Randolph	1 / 0 / 13	0 / 0 / 0	2 / 1 / 0	17
Reiner Knizia	1 (1) / 6 / 17	0 / 0 / 1	1 / 4 / 5	36
Marie Fort	0 / 0 / 0	0 / 0 / 0	2 (1) / 1 / 0	4
Wilfried Fort	0 / 0 / 0	0 / 0 / 0	2 (1) / 1 / 0	4
Bruno Cathala	1 (1) / 0 / 7	0 / 0 / 1	1 / 0 / 0	11
Heinz Meister	0 / 1 / 2	0 / 0 / 0	2 / 3 / 5	13
Alan R. Moon	2 / 2 / 5	0 / 0 / 0	0 / 0 / 0	9
Manfred Ludwig	0 / 0 / 1	0 / 0 / 0	2 / 2 / 1	6
Antoine Bauza	1 / 0 / 2	1 / 0 / 1	0 / 1 / 0	6
Andreas Seyfarth	2 / 1 / 1	0 / 0 / 0	0 / 0 / 0	4
Klaus Zoch	0 / 1 / 1	0 / 0 / 0	2 / 0 / 0	4
Andreas Pelikan	0 / 1 / 0	2 / 0 / 0	0 / 0 / 0	3
Alexander Pfister	0 / 0 / 0	2 / 0 / 2	0 / 0 / 0	4
Steffen Bogen	1 / 0 / 0	0 / 0 / 0	1 / 0 / 1	3
Peter-Paul Joopen	0 / 0 / 0	0 / 0 / 0	2 / 0 / 1	3
Donald X. Vaccarino	2 / 0 / 0	0 / 0 / 0	0 / 0 / 0	2
Vlaada Chvátil	1 (1) / 0 / 2	0 / 0 / 0	0 / 0 / 0	4
Rüdiger Dorn	0 / 5 / 3	1 / 0 / 0	0 / 1 / 0	10
Wolfgang Warsch	0 / 1 / 0	1 / 1 / 0	0 / 2 / 0	5
Kai Haferkamp	0 / 0 / 0	0 / 0 / 0	1 / 3 / 9	13
Marco Teubner	0 / 0 / 1	0 / 0 / 0	1 / 3 / 6	11
Jens-Peter Schliemann	0 / 0 / 2	0 / 0 / 0	1 / 2 / 3	8
Roberto Fraga	0 / 0 / 1	0 / 0 / 1	1 / 2 / 1	6
Bernhard Weber	0 / 0 / 1	0 / 0 / 0	1 / 1 / 2	5
Michael Schacht	1 / 0 / 8	0 / 0 / 0	0 / 0 / 1	10
Sid Sackson	1 / 0 / 7	0 / 0 / 0	0 / 0 / 0	8
Rudi Hoffmann	1 / 0 / 6	0 / 0 / 0	0 / 0 / 0	7
Günter Burkhardt	0 / 0 / 5	0 / 0 / 0	1 / 0 / 1	7
Dirk Henn	1 / 0 / 5	0 / 0 / 0	0 / 0 / 0	6
Matt Leacock	0 / 3 / 0	0 (1) / 1 / 0	0 / 0 / 0	5
Uwe Rosenberg	0 (1) / 1 / 4	0 / 0 / 2	0 / 0 / 0	8
Jürgen P. Grunau	0 / 2 / 0	0 / 0 / 0	0 / 3 / 0	5
Stefan Dorra	0 / 2 / 6	0 / 0 / 0	0 / 1 / 1	10
Stefan Feld	0 / 0 / 5	0 / 3 / 1	0 / 0 / 0	9
Leo Colovini	0 / 2 / 5	0 / 0 / 0	0 / 1 / 0	8
Wolfgang Dirscherl	0 / 0 / 0	0 / 0 / 0	0 / 2 / 4	6
Reinhard Staupe	0 / 0 / 7	0 / 0 / 0	0 / 1 / 2	10
Friedemann Friese	0 / 1 / 3	0 / 0 / 1	0 / 0 / 0	5
Reinhold Wittig	0 / 0 / 10	0 / 0 / 0	0 / 0 / 1	11
Kris Burm	0 / 0 / 6	0 / 0 / 0	0 / 0 / 0	6
Roland Siegers	0 / 0 / 6	0 / 0 / 0	0 / 0 / 0	6
Haim Shafir	0 / 0 / 0	0 / 0 / 0	0 / 0 / 6	6
Jacques Zeimet	0 / 0 / 5	0 / 0 / 0	0 / 0 / 0	5

Download the full results here: designers and games. As always, you can find the code for this analysis from GitLab.

Arguably, there’s 15 Kennerspiel des Jahres winners: In 2017, the jury gave the award to the first three games in the Exit series. In all statistics, I only count them as a single entry. Similarly, in 2019, the first three entries of the Sherlock series got recommended, which I also count as just a single game. ↩︎

Reverse engineering the BoardGameGeek ranking – Part 2!

contact@recommend.games (Recommend.Games) — Tue, 26 Jan 2021 22:24:00 +0200

This is the second part of a series explaining and analysing the BoardGameGeek rankings. Read the first part here.

Last time I left you with the nice result that BoardGameGeek (BGG) calculates its ranking by taking users’ ratings for a particular game and then add around 1500-1600 dummy ratings of 5.5. This so-called geek score is used to sort the games from best (Gloomhaven) to worst (Tic-Tac-Toe).

One detail however we touched on in passing, but did not resolve, is how that number of dummy ratings develop over time. When the current calculation method was introduced, BGG founder Scott Alden mentioned that this number would be pegged to the number of total ratings, but did not reveal any details. Challenge accepted!

In order to tackle this question, we need to compare that dummy number to the total number of ratings over time. Fortunately, thanks to the scraping done for Recommend.Games, we have access to the BGG games data over the past year or so. Using these snapshots, we observe how the number of games and ratings in the database has grown:

We can now repeat the exact same calculation we did in the previous post: For each point in time, the algorithm searches for the number of dummy ratings that yields an estimated geek score closest to the actual score. Now, we have a bunch of data points that correlate the total number of ratings with the number of dummies used at that time. Here’s what it looks like:

We get a pretty nice straight line – the dashed line in the plot is fitted with linear regression, i.e., the straight line that most closely fits our data. Its formula is:

\[ \textrm{number of dummies} \approx 0.0000997 \cdot \textrm{total number of ratings}. \]

This means that for every rating entered into the BGG database, the number of dummy ratings is increased by 0.0000997. That number might look a bit opaque, but it’s actually very easy to interpret once you put the question to its head: How many ratings have to be entered for the number of dummies to increase by 1? You get the answer to that by taking the inverse of that factor, which happens to be about 10,032. This number is way to close to 10,000 to be a coincidence! We can conclude the exact formula for the number of dummy ratings:

\[ \textrm{number of dummies} = \frac{\textrm{total number of ratings}}{10\,000}. \]

As of the time of writing, there are 17,287,904 ratings (give or take) in the BGG database, so there will be around 1729 dummy ratings of 5.5 added to the regular ratings.

As the number of BGG users rises steadily, this number of dummy ratings also keeps increasing. This is part of the reason why older games (particularly those with a newer edition) tend to drop in the rankings. When users stop adding new ratings, a game’s average rating more or less freezes. But because more and more dummy votes are added, the geek score decreases every time it gets recalculated, and so the older games drop in the rankings, while the latest hotness gets all the fresh votes, and shoots up to the top.

The circle of hype.

Reverse engineering the BoardGameGeek ranking

contact@recommend.games (Recommend.Games) — Sat, 03 Oct 2020 08:42:51 +0300

TL;DR: BoardGameGeek calculates its ranking by adding around 1500-1600 dummy ratings of 5.5 to the regular users’ ratings. They called it their geek score, statisticians call it a Bayesian average. We use this knowledge to calculate some alternative rankings.

I often describe BoardGameGeek (BGG) as “the Internet Movie Database (IMDb) for games”. Much like its cinematic counterpart, the biggest board game database not only collects all sorts of information obsessively, but also allows users to rate games on a scale from 1 (awful - defies game description) to 10 (outstanding - will always enjoy playing). These ratings are then used to rank games, with Gloomhaven occupying the top spot since December 2017.

While BGG founder Scott Alden admitted in a recent interview on the excellent Five Games For Doomsday 🗄️ podcast that he doesn’t care all that much about the rankings, gamers around the world certainly do. They would discuss heatedly any movement in the rankings, question why games X is up there while game Y is missing, and generally criticise the selection for either having too many or not enough recent releases.

Reason enough for me to take a closer look at how the rankings work and some of the maths behind it.

Generally speaking, we want to rank a game higher the better its score is. The first instinct would be to just sum up all the ratings users gave to that particular game, divide by the number of ratings, and rank games from highest to lowest. What I just described would be the arithmetic mean (or just average if you feel less fancy) of the ratings, which is simple and intuitive, but suffers from a sever defect: a game with a single rating of 10 would always sit on top of the ranking, well ahead of much beloved games with thousands of votes that couldn’t possibly be all 10s.

The easiest fix is filtering out games with less than a certain number of ratings, say 100.¹ That’s a decent enough approach, and yields the following top 5 games as of the time of writing:

Notably, those are all very recent games with relatively few ratings.² Some might consider this a feature, not a bug, but when your intention is to create a list of the best board games, you probably do want to give a nod to proven classics, and not just the latest hotness. How to balance out these ends of the spectrum is in the end a choice you have to make, and no matter what it is, People on the Internet™ will not like it.

The way both IMDb and BGG chose to tackle this issue is by essentially not trusting the ratings – at least not too much. The method boils down to assigning a new item in the database (be it movie or game) a predefined average by default, and only gradually trusting the ratings’ average as thousands and thousands of users have cast their votes. More concretely the rankings are calculated by adding a number of dummy ratings with a chosen average value, say 5.5, to each game’s regular ratings. The result is that initially each game will have a score close to 5.5, but as more users rate the game, that score will move closer and closer to the conventional mean.

BGG calls this their geek score. Mathematically speaking, it is a Bayesian average, and calculates as follows:

\[ \textrm{geek score} = \frac{\textrm{sum of ratings} + \textrm{number of dummies} \cdot \textrm{dummy value}}{\textrm{number of ratings} + \textrm{number of dummies}}, \]

where sum of ratings can be calculated either by, well, summing up all ratings or via number of ratings ⋅ average rating. Don’t worry too much about the details though – adding dummy ratings is really all you need to understand.

OK, so that’s the concept, but crucially that’s not all the details. You still need to choose how many dummy ratings you want to add and what value they should take. Since People on the Internet™ who disagree with your ranking will try to manipulate it in whatever way they can, sites are usually very cagey about said details. IMDb used to be more transparent, as was BGG, but now we have to dig a little deeper.

Let’s start from the easier of the two, the value of the dummy ratings. It is commonly chosen to represent some prior mean, i.e., some decent estimate of the rating a new game in the database would have. A frequent choice would be to use the average rating across all games. It’s a fair assumption – without further information about a game, we don’t know if it’s any better or worse than the average game. However, Scott Alden actually gave away the answer in that interview from the beginning: BGG chose the dummy value to be 5.5. Their rationale is that ratings range from 1 through 10, so 5.5 is the midpoint. Of course, people tend to rather play and rate much more the games they like, and so the average rating is around 7. Opting for the lower value here is part of the design of the ranking: it means a new game would enter the ranking rather at the end of the pack. On the other hand, using the mean as the dummy value means a new game is placed more or less in the middle. It is worth mentioning that IMDb does use the mean (or at least used to), but they only ever publish the top 250 movies, and don’t care about the crowd behind.

The other value, the number of dummy ratings, requires more work. Because some of the details and data are unknown, we cannot actually pin down the exact number that BGG is using. Instead, we’ll try three different approaches, and compare their results.

Formula

On the surface, this should be super easy to solve: in the formula above, we know every single value but the number of dummy ratings. BGG publishes the number of ratings, their arithmetic mean, and the “geek score” or Bayesian average for every game, and we know that the dummy value is 5.5. With a little high school algebra we solve the formula for number of dummies:

\[ \textrm{number of dummies} = \textrm{number of ratings} \cdot \frac{\textrm{average rating} - \textrm{geek score}}{\textrm{geek score} - \textrm{dummy value}} \]

Now we should be able to plug in those values for any given game, say Flamme Rouge, and get the result. With 10,936 ratings that average 7.562, and a geek score of 7.266, this yield:

\[ \textrm{number of dummies} = 10936 \cdot \frac{7.562 - 7.266}{7.266 - 5.5} \approx 1830. \]

So, there’s about 1830 dummy ratings, end of story. Right? Unfortunately, not quite. When computing this formula for different games, the results vary wildly, as you can see from this histogram over the results for the same calculation with other games:

And this plot is even cropped, the results vary from -1.4 million to +810 thousand, though some 90% lie within the above range, with a mean of around 1604 and a median of around 1590.

What’s going on, why are the results so inconsistent? The problem is the ranking’s secret sauce. Both IMDb and BGG stress is that they only consider regular voters for their rankings. That’s the most mysterious part of the system as it’s the easiest to manipulate, so we’ll just have to take their word for it. For this investigation it means that the average rating BGG publishes includes all the ratings, but the geek score might not.

Still, clearly something is happening around the 1600 ratings mark, so we are at least getting closer to an answer. If exact calculations won’t work, maybe we can approximate the correct value instead?

Trial & error

Let’s take a step back here. What we’re really trying to achieve here is not finding the exact formula for that mysterious “geek score”, but rather recreate the BGG ranking. That is, we want to find the values in the above formula, such that the resulting ranking matches BGG’s ranking as closely as possible. Luckily, statistics has all the tools we need. Spearman correlation measures rank correlation – just what we need. This will be 1 if both rankings sort in exactly the same way, 0 if there’s no relation, and -1 if they sort exactly the opposite way. Again, don’t worry about the details, just trust the maths.

What we can do now is fairly simply and quickly compute the rankings for different number of dummy ratings, and pick the value with the highest Spearman correlation. Without further ado, here are the results:

The best correlation of around 0.996 is achieved with 1488 dummy ratings. However, it is worth noticing that the changes in the correlation are very, very small throughout the range we examined here (1000 to 2500), so let’s dig still a little deeper.

Optimisation

What we have here at hand is actually a classic optimisation task: a real valued function in one unknown (or two if we allow a variable dummy value as well) which we’d like to maximise. This is a well-studied field, with many fast and simple implementations that provide us the solution in no time. Unsuprisingly, we get the same result as above: the best possible correlation is 0.996 with around 1488 dummy ratings.

But since we made it this far, let’s take it one step further. So far, we tried to optimise the correlation in order to recreate BGG’s ranking. However, we can also try to recreate the actual geek scores. That is, we can look for the number of dummy ratings that will yield the closest to the actual geek score with our calculations. What exactly we mean by “closest” is up to us to define. A common metric is the mean squared error.³ It’s not worth getting into the maths here either, but the general idea is that we want to punish outliers in our estimates more (qudratically so) the further away they lie from the actual datapoint. Long story short, this yields a minimum for around 1636 dummy ratings.

Let’s take one last swing and see what happens if we don’t fix the dummy value at 5.5 but allow that to be variable as well. This is no problem for the optimisation algorithm and yields the following results:

the best correlation with 1942 dummy ratings of 5.554, and
the least squared error with 1616 dummy ratings of 5.494.

Either of those improvements in the performance metrics are hardly noticable (in fact insible after rounding), but they do confirm nicely a dummy value of 5.5.

Conclusion

All things consider, we can be confident that BoardGameGeek calculates their rankings by adding around 1500 to 1600 dummy ratings of 5.5 to the regular users’ ratings. What exactly constitutes a regular user, and what ratings might be discarded due to shilling, remains a well guarded secret though. Note that the number of dummies is pegged to the overall number of ratings, so this is a moving target, and the calculations would change as time passes.

Now I must applaud anybody who actually made it all the way through this pretty dry and technical article. The real reason why I dwelled so much on the ratings, and how they are compressed into the BGG rankings, is to get a feeling of what’s going on behind the scenes, what the can express, and what they cannot or even do not try to express. Another major take-away is that any of these decisions are choices that need to be made and that come with certain tradeoffs – like them or not.

In this particular case, I have the feeling that both the Cult of the New™ and connoisseurs of classic games are equally unhappy about the BGG top 100, which one should probably consider a compliment.

Alternative rankings

I’ll send you off with some rankings that were obtained by making different choices for the two values that we discussed throughout this article: the number of dummy ratings and their value.

Using the ratings average as dummy value

I’ve mentioned before that the average rating across all games is around 7 – a little¹ more precisely 7.08278. What if we chose that as the dummy rating, but left their number at 1600? The result should be a ranking that is a little friendlier to newer titles with fewer ratings as their score isn’t dragged all the way down to 5.5 in the beginning.

Sure enough, the brand new Jaws of the Lion with less than 3000 ratings already shows up in the top 10. The other game that sticks out here is Kingdom Death: Monster. This Kickstarter success story clearly attracted a lot of enthusiasts, but not necessarily the mass.

Using the top 250 number of ratings

Just like IMDb publishes only their top 250 movies, we can consider the same and crank up the number of dummy ratings. A good number seems to be the 250th most rated game on BGG, which has been rated 12,014 times. Using BGG’s standard dummy value of 5.5, we obtain a ranking that is much more skewed towards proven classics:

The most recent release on this list is Gloomhaven, but we also meet again old BGG #1’s: Puerto Rico and Agricola.

Combining both!

Finally, let’s do what IMDb does (or used to do), and add to each game’s ratings 12,014 dummy ratings of 7.08278:

The effects of more, but higher dummy ratings seem to almost cancel each other out. Compared to BGG’s actual top 10, only Twilight Imperium and Gaia Project are missing, otherwise this ranking looks very familiar. Turns out, BGG did a pretty good job designing its ranking!

Read the second part here where we nail down the number of dummy ratings to the dot.

PS: You can find the notebook I used to do all the calculations on Kaggle.

PPS: Turns out that GoDataDriven did almost the same calculation 🗄️ three years ago – even with the same title! Back then, they estimated that BGG added 725 dummy ratings. Jorge Nieva replicated their analysis with a more recent dataset, which thankfully yielded a very close match to our result: 1594 dummy ratings.

Throughout this article I only considered games with at least 100 ratings, mostly to ensure that the very long tail of games with few ratings won’t unduely skew the results. However, most of the calculations would only change in some negligible decimals when including all games. ↩︎ ↩︎
Jaws of the Lion is something of an exception here and will undoubtably shoot into the BGG top 10 very soon. In fact, it might be the only game with the potential to unseat Gloomhaven as the number 1. ↩︎
It’s probably even more common to use the root mean squared error, but for boring mathematical reasons, it doesn’t make a difference when it comes to optimisation. In fact, we could even drop the word mean from our metric and still obtain the same optimal point, so let’s not dwell on this. ↩︎