Machine Learning on Analysis Paralysis

Teaching Elo to Play with Friends

contact@recommend.games (Recommend.Games) — Fri, 12 Dec 2025 12:00:00 +0200

At some point this year, I let my laptop run flat-out for almost two weeks just to answer one question: how much of a four-player board game is “skill” and how much is “luck”? That sounds excessive, but there was a catch: before I could even start those simulations, I had to fix a basic problem. Elo – the rating system we’ve been happily using so far – only really knows how to handle one-on-one duels.

This article is the missing technical chapter in the series. In part 1 we met Elo and learned how it turns match results into skill ratings. In part 2 we sent those ratings to the Crucible to predict the next World Snooker Champion. And in part 3 we stole a clever idea from Dürsch, Lambrecht and Oechssler to turn the spread of Elo ratings in a two-player “toy universe” into a kind of skill-o-meter: a way to say whether a game behaves more like a 30%-skill world or an 80%-skill world.

There’s one obvious gap left: most modern board games aren’t tidy head-to-head affairs. Around a real table you’ll usually find three, four, sometimes five players battling it out in CATAN, Brass, Gaia Project or whatever your current obsession is. If we want to use our shiny skill-o-meter on those games, we first have to teach Elo how to cope with real multiplayer tables instead of just faking them as a stack of two-player matches.

Fair warning: this part is even more technical than part 3. We’ll talk about probability matrices, permutations and a scary-looking formula or two. If that’s not your thing, you’re still very welcome to skim the maths-heavy bits – I’ll keep pointing out the important intuitions along the way. The payoff is worth it: by the end of this article, we’ll have a principled multiplayer Elo system and a checked-and-calibrated skill-o-meter that still works when three, four or five people sit down to play.

Why two-player Elo isn’t enough for modern games

Elo’s original paper was targeted at chess, so naturally it was only concerned with two-player games. Likewise, everything I’ve talked about in this series so far has assumed a simple head-to-head match: one player vs another, winner takes the Elo chips.

If we want to apply our shiny “skill-o-meter” from part 3 to the games we actually play, we need to teach Elo how to handle true multiplayer tables instead of just faking them as a bunch of two-player matches.

How people fake multiplayer Elo (and why it’s not quite right)

If you’re like me and spend a slightly embarrassing amount of your free time on Board Game Arena, you might have noticed their Elo implementation. They simply treat multiplayer games as a collection of 1‑vs‑1 battles. So if Alice, Bob and Carol play a game, their Elo calculations treat this as three matches: Alice vs Bob, Alice vs Carol and Bob vs Carol. If Alice indeed won the game, Bob came in second and Carol last, Alice would win both her “virtual” matches and Bob his against Carol. Elo ratings would then be updated according to the regular formula, with $K$ “adjusted for player count” (I didn’t find an up-to-date source as to the details).

Conceptually, this is a neat hack but not quite right: it pretends Alice actually played two independent duels against Bob and Carol, even though in reality all three interacted in the same shared game state and their decisions affected each other at the same time.

Note that for an $n$-player game there are ${n \choose 2} = \frac{n(n-1)}{2}$ pairings, so the number of updates grows quadratically with player count. This kind of growing complexity can really come back to bite you in the behind when it comes to compute, but (a) luckily we don’t need to worry about matches with hundreds of players in tabletop gaming and (b) it could be much worse, as we shall see in a minute…

A more principled multiplayer Elo: ranking probabilities

In part 3, we already leaned on a neat idea by Peter Dürsch, Marco Lambrecht and Jörg Oechssler from their paper “Measuring skill and chance in games” (2020). There we used their framework to turn the spread of Elo ratings into a “skill-o-meter” for two-player games. In this article, we’re going back to the same well: DLO also propose a way to run Elo on proper multiplayer tables, and that’s exactly the tool we need for modern board games.

From table results to expected payoffs

Dürsch et al suggest a more principled way to deal with multiplayer tables. Let $n$ be the number of players in the match. Instead of pretending everyone played everyone else in separate duels, they directly model the whole finishing order at once.

The first ingredient is an $n\times n$ matrix of probabilities:

\[ p_{ij} = P(\text{player $i$ finishes in position $j$}). \]

You can read row $i$ as “what’s the chance player $i$ finishes 1st, 2nd, …, last?” and column $j$ as “who is most likely to end up in position $j$?”.

Just like in the two-player case, we need a numerical payoff to compare expectations with reality. For an $n$-player game we give the winner $n-1$ points, the runner-up $n-2$, all the way down to 0 for last place.¹ If there are ties, we give each tied player the average of the payoffs they straddle. That gives us the expected payoff for player $i$:

\[ e_i = E[\text{payoff for player $i$}] = \sum_{j=0}^{n-1} p_{ij} (n - 1 - j). \]

Once we have that, the Elo update looks exactly like before. Let $a_i$ be the actual payoff (from the final ranking, scaled in the same way). We compare $a_i$ to $e_i$, and shift the rating in the direction of the surprise:

\[ r_i \leftarrow r_i + \frac{K}{n-1} (a_i - e_i). \]

The factor $1/(n-1)$ just normalises things so that one whole game still corresponds to about $K$ “chips” moving around, as in the two-player version.

From Elo ratings to ranking probabilities

This is where things get a bit heavy. If you’re mostly here for the big picture, feel free to skim or even skip the formulae in this section — I’ll summarise the important part again at the end.

Conceptually, what we want is simple: for a given set of Elo ratings, we assign a probability to each possible finishing order of the players. Stronger players should be more likely to end up near the top, weaker ones near the bottom. Once we have those probabilities, we can add them up to get the chance that a particular player finishes in a particular position.

Formally, we write a possible ranking as a permutation $\tau$ of ${0, \dots, n-1}$, where $\tau(j)$ tells us which player ends up in position $j$ (with $j=0$ for the winner, $j=1$ for second place, and so on). The probability of seeing a particular ranking $\tau$ can be written using the chain rule of probability:

\[ P(\tau) = P(\text{players $\tau(0), \dots, \tau(n - 1)$ on positions $0, \dots, n - 1$}) \\ = \prod_{j=0}^{n-1} P(\text{player $\tau(j)$ on position $j$} \mid \text{players $\tau(0), \dots, \tau(j - 1)$ fixed above}). \]

To estimate those conditional probabilities, Dürsch et al use the softmax over Elo ratings. Softmax is just the multiplayer cousin of the Elo win-probability formula: you take a “strength score” for each player, exponentiate it, and then divide by the sum so that everything adds up to 1. At each step $j$, we look at the players who haven’t been placed yet and assign probabilities proportional to $10^{r / 400}$, just like in the two-player Elo formula. If we write $r_i$ for the current rating of player $i$, this gives:

\[ P(\text{player $\tau(j)$ on position $j$} \mid \text{players $\tau(0), \dots, \tau(j - 1)$ fixed above}) \\ = \frac{10^{r_{\tau(j)} / 400}}{\sum_{k=j}^{n-1} 10^{r_{\tau(k)} / 400}}. \]

Plugging this into the chain rule expression yields a compact formula for the probability of a full ranking $\tau$:

\[ P(\tau) = \prod_{j=0}^{n-1} \frac{10^{r_{\tau(j)} / 400}}{\sum_{k=j}^{n-1} 10^{r_{\tau(k)} / 400}}. \]

Now, to get the entries of our probability matrix, we just have to sum over all rankings that put a given player in a given position. Remember that $p_{ij}$ is the probability that player $i$ finishes in position $j$. With the convention $\tau(j) = i$ meaning “player $i$ sits in position $j$”, we have:

\[ p_{ij} = \sum_{\tau \text{ with } \tau(j) = i} P(\tau). \]

If the formulae lost you at some point, that’s OK — the story is simple: we assign a probability to each possible finishing order based on the Elo ratings, and then sum those probabilities to find out how likely each player is to end up in each position. That’s all you really need to remember from this section.

Does this really generalise two-player Elo?

You might still wonder if it’s really justified to call this a generalisation of two-player Elo, since it looks rather different at first glance. The crucial sanity check is that when we only have $n = 2$ players at the table, all of this machinery collapses back to the usual head-to-head model: there are only two possible rankings, the probability matrix reduces to the familiar win–loss probabilities, the payoff vector $(1, 0)$ just scores win vs loss, and the update rule becomes exactly the original Elo formula again.² You don’t need to wade through the algebra – the important point is that for ordinary two-player encounters, this system behaves just like classic Elo.

The price of doing it properly: combinatorics and compute

There is one big catch we’ve glossed over so far. To calculate the entries of the probability matrix $p_{ij}$, we have to sum $P(\tau)$ over all possible rankings $\tau$. If you remember your combinatorics basics, you’ll know that there are $n!$ permutations of $n$ players – a function that grows even faster than exponential. In other words: a straightforward implementation of this model is computationally very expensive.

Does this mean the whole approach is doomed? Luckily, not quite. Most board games have at most five or six players, and $6! = 720$ is big but still perfectly manageable on a modern computer. That covers the vast majority of situations we care about in tabletop gaming.

For higher player counts there are more efficient tricks (for example dynamic programming and Monte Carlo approximations) that avoid looping over all permutations explicitly. I’m not going to go into the details here; if you’re curious, you can have a look at the implementation in the code for this article – but for our purposes it’s enough to know that the full model is tractable for realistic games.

Multiplayer p-deterministic games

Right, after so much theory you deserve something a bit more concrete. Real-world applications will come in the next article; for now, there’s still one more thing to check: do the multiplayer versions of the $p$-deterministic game behave in the same way as the two-player toy world we built in part 3?

The setup remains almost the same. We fix an underlying skill ranking for all players. For each game, we flip a weighted coin: with probability $p$ we play a game of pure skill, where players finish in order of their underlying strength; with probability $1-p$ we play a game of pure chance, where the finishing order is just a random permutation of the players. It’s the same toy universe as before, just with more than two players sitting at the table each time.

The σ vs p benchmark still holds for up to 15 players

With this multiplayer version of the $p$-deterministic game in hand, we can run the same kind of simulations as before. For each choice of $p$ and for player counts between 2 and 15, we let lots of games play out, calibrate $K$ on the simulated match data, compute the resulting Elo ratings and record their standard deviation $\sigma$. Plotting $\sigma$ against $p$ for each player count gives us this family of curves:

All of these curves are smooth and strictly increasing: as we turn up $p$ and let skill matter more often, the Elo spread $\sigma$ grows, just like in the two-player case. More interestingly, when we plot these player counts from 2 up to 15, the points for different player counts are essentially indistinguishable: for each value of $p$, all the coloured dots sit almost exactly on top of each other. Any tiny visible wobble at very high $p$ is well within the limits of simulation noise and numerical quirks.

That’s precisely the behaviour we were hoping to see. Empirically, in this toy universe $\sigma$ is effectively a function of $p$ alone and — within our numerical precision — invariant to how many players sit at the table, even up to 15. In practical terms, this means that if we measure a standard deviation $\sigma$ in a real three-, four- or five-player game, we can safely read off a corresponding “$p$-skill world” from this benchmark without worrying about the exact player count.

Talking of the computational effort: getting this last plot alone down to only about two weeks of wall-clock time on my poor laptop took a fair bit of optimisation. The result might look a little underwhelming after all that build-up, but that’s exactly the point: after grinding through all those simulations, the curves stubbornly agree that player count basically doesn’t matter. 🔥😅🤓

Where this leaves us (and what’s next)

We’ve covered a lot of ground in this article, but the payoff is twofold.

First, we now have a principled way to run Elo on real multiplayer tables. Instead of faking CATAN or Brass as a pile of head-to-head duels, we can model the whole finishing order at once, get sensible expected payoffs for each seat, and update ratings in a way that reduces to classic two-player Elo when there are only two people at the table.

Second, we’ve stress-tested our “Elo-as-a-skill-o-meter” from part 3 in a richer toy universe. In those $p$-deterministic worlds, the standard deviation $\sigma$ of Elo ratings turns out to depend almost entirely on $p$ and, within numerical accuracy, not on how many players sit down to play. That means $\sigma$ really does behave like a calibrated skill dial we can use for 2–6 player games.

Put together, this gives us the toolset we wanted: given real multiplayer game logs, we can (a) fit Elo using the multiplayer update, (b) calibrate $K$ on predictive accuracy, (c) read off the resulting $\sigma$ and map it to a “skill fraction” $p$ using our benchmark curve.

Next time, we’ll finally unleash this machinery on actual board games. We’ll look at real play logs, see which games behave more like 30%-skill worlds and which ones look closer to 80% skill, and maybe settle a few pub arguments along the way. 🤓

Dürsch et al use a flexible payoff structure which makes the formulae and implementation more confusing. For our purposes, the fixed payoff based on ranks is enough, so I tried to keep things simple. ↩︎
If you’re itching to do the algebra yourself, be my guest — that’s the unofficial “exercise to the reader” for this section. I decided you didn’t need to watch me juggle minus signs for a page. ↩︎

Elo as a Skill-O-Meter

contact@recommend.games (Recommend.Games) — Fri, 05 Dec 2025 12:00:00 +0200

Whether a game counts as “skill” or “chance” isn’t just a pub argument — in many countries it’s a legal distinction. Roulette and blackjack live on the “chance” side; tennis and chess are filed under “skill”. Different rules, different taxes, different ways for people to lose money.

The trouble is that this line is usually drawn by tradition and gut feeling. Is poker really “more skill” than backgammon? Is snooker closer to roulette or closer to chess? A group of economists tried to answer that question more systematically: instead of arguing, measure how “skill-heavy” a game is in practice by looking at the Elo ratings of all its players. We’ll meet their work properly in a bit.

In this article I want to steal that idea for board games. So far we’ve used Elo to track individual player strength; this time we’ll go one level up. Instead of asking who is strong, we’ll look at the whole distribution of Elo ratings in a game and see what its spread can tell us about luck and skill — turning Elo into a kind of “skill-o-meter”.

From Elo ratings to skill distributions

By now the basics of Elo should be familiar: each player gets a rating that reflects their playing strength, rating differences go into a simple formula to give expected win probabilities, and after each match we update those ratings based on whether players beat expectations. If you want the full story (including all the maths and the logistic regression detour), part 1 has you covered; here we’ll treat Elo as a black box that turns match results into reasonable estimates of player skill.

In part 2 we applied this system to predict the 2025 World Snooker Champion. The model’s favourite, John Higgins, didn’t manage to win his fifth title, but it did give eventual winner Zhao Xintong a 10.6% chance when the bookies only gave him 5.9%. I’ll take that as a personal win — and as evidence that Elo isn’t just numerology, it really does capture something about players’ skills.

Wider distributions, more skill

What we’re really after is a way to say how much skill is involved in a game, and to compare different games with each other. The individual Elo numbers are just a stepping stone. To get anywhere, we have to zoom out and look at the whole distribution of skills, as measured by all players’ Elo ratings.

The basic idea is simple: if a game is mostly luck and players’ decisions don’t matter much, nobody can reliably stay ahead for long. You’ll see some winning streaks, but they’ll wash out again, and everyone’s ratings will cluster around 0. In a game where skill really matters, the strongest players win more often than they lose and slowly drift away from the pack. The result is a much wider Elo distribution: a long tail of very strong players and a long tail of weaker ones.

A first look: snooker vs tennis

Let’s make this more concrete. We’ve already calculated Elo ratings for snooker, so let’s compare it to another English upper-class sport played on a green surface: tennis. Can you tell which one is more skill-based? To get a more objective answer, we look at the Elo distributions for both and see which one is wider:

According to this plot, the Elo ratings of snooker players have a much higher peak and shorter tails, which suggests that outcomes are more influenced by luck. Tennis — at least on the WTA, the women’s tour¹ — seems to show a wider spread and therefore more room for skill. But how do we know these distributions are even comparable? And can we turn that vague “more luck, more skill” into an actual number?

Turning spread into a skill measure

To answer these questions, we need to properly dive into the science. 🧑‍🔬

The one neat trick economists use to measure skill (that gamers can steal)

I’m going to lean on a neat idea by Peter Dürsch, Marco Lambrecht and Jörg Oechssler, from their paper “Measuring skill and chance in games” (2020). They come from an economics background and originally cared about gambling regulation, but the trick itself is much more general: take the Elo ratings for all players in a game, look at their distribution, and from that pin down a single number that tells you where the game sits on the spectrum between “pure chance” and “pure skill”. That’s exactly what we’re trying to do here — just for board games instead of casinos. 🤑

So we’ll follow their lead and focus on the spread of the Elo distribution as our measure of how much skill shows up in a game.

Standard deviation of Elo ratings

The mathematical measure for the spread of a distribution is its standard deviation $\sigma$. The wider the distribution, the larger its standard deviation. Roughly speaking, it’s the expected (squared) difference from the mean. In our setting, that means $\sigma$ tells us how far, on average, players’ skills lie from the “average” player: a bigger $\sigma$ means the field is more spread out, with larger typical gaps between players — exactly the sort of quantity we want to look at.

So from now on, whenever I talk about the “amount of skill” we see in a game, I’ll use the standard deviation of its Elo ratings, $\sigma$, as the proxy.

The problem with K

There’s an important caveat: Elo ratings and their distribution crucially depend on $K$, the update factor. If you remember the metaphor from the first article, $K$ is the number of “skill chips” the players put into the pot each game. Higher stakes lead to a wider spread; if almost no chips change hands, everyone ends up with roughly the same number and the spread stays tiny.

A natural idea would be to just fix one $K$ for all games. Unfortunately, that doesn’t work either. Imagine two ladders for exactly the same game with the same population of players. In ladder A, everyone plays a handful of games per year; in ladder B, the same people grind hundreds of games a month. We now run Elo with the same $K$ on both datasets. In ladder A only a few “skill chips” ever change hands before the season is over, ratings barely have time to drift apart, and the final distribution stays fairly tight. In ladder B, chips slosh back and forth for thousands of rounds; random streaks get amplified, the system has time to separate the strong from the weak, and the rating spread ends up much wider. The underlying game and the underlying skills are identical, yet the dispersion of Elo ratings depends heavily on how often people play and how long we observe them. Fixing $K$ globally doesn’t make the spread an intrinsic property of the game — it just bakes in arbitrary design decisions about volume and time.

Calibrating K from the data

What Dürsch et al suggest instead is to calibrate $K$ from the data, so that the Elo ratings are as good as possible at the job they were designed for: predicting who wins. Remember how Elo works: we take the pre-match ratings $r_A$ and $r_B$ of two players A and B, and feed the difference into a logistic formula to get the predicted win probability for A:

\[ p_A = \frac{1}{1 + 10^{-(r_A - r_B) / 400}}. \]

After the match, we compare this prediction with $s_A$, the actual outcome of the match ($s_A = 1$ if A won, $s_A = 0$ if they lost and $s_A = 0.5$ in case of a tie), and update:

\[ r_A \leftarrow r_A + K (s_A - p_A). \]

The basic assumption in DLO’s approach is that $K$ is “optimal” if these prediction errors $s_A - p_A$ are, on average, as small as possible. For a given $K$ and a set of matches $t \in {1, …, T}$, we look at the squared errors and minimise their mean:

\[ K^* = \argmin_K \frac{1}{T} \sum_{t=1}^T (s^{(t)} - p^{(t)})^2. \]

This mean squared error is a standard loss for training machine learning models; when we apply it to probabilistic predictions like this, it’s known as the Brier loss. We can search over $K$ to find $K^*$, the update factor that makes Elo’s predictions as accurate as possible on that dataset.² Different games (and different datasets) will generally end up with different $K^*$. Once we’ve found $K^*$, we can run Elo with that value and then look at the resulting rating distributions.

Why K* is not our skill metric

Before we happily run Elo with $K^*$ and stare at the resulting distributions, let’s pause and ask what $K^*$ itself is telling us. Earlier I compared $K$ to a step size in an iterative learning process: a larger $K$ means we take bigger steps on each update and let a single match pull the ratings around more. If the “optimal” $K^*$ is large, doesn’t that mean each game is very informative about the players’ skills? That sounds suspiciously close to what we’re trying to measure. Can we just use $K^*$ as our coveted luck–skill number?

Not quite. First of all, as we’ve already discussed above, the optimal $K$ depends strongly on the player population. Larger sets of matches will tend to have smaller $K^*$, even if the underlying skill levels are exactly the same, simply because with more data you don’t need to react as violently to each individual result. That’s why we have to calibrate $K^*$ on the exact dataset we’re using.

Second, two games might demand the same underlying skills, but still have very different learning curves: some are slow and steady, others click after a single “epiphany”. Those learning dynamics also feed into $K^*$: in a game where people improve in big jumps, you’ll see a different “optimal” step size than in a game where everyone creeps up gradually. So even if two games are equally skill-based in the end, their $K^*$ values can be quite different, and comparing them would be misleading.

Luckily, the standard deviation of the Elo distribution is much more robust to those issues than $K^*$ itself: it mostly cares about where everyone ends up, not about how fast they got there.

We now have the theoretical foundation to compute Elo distributions and their standard deviation. What we still need is actual game data. I’ve already teased how this applies to snooker and tennis, and in a later article we’ll look at many more concrete examples.

A toy universe of luck and skill

Before we get there, though, I want to take a closer look at a synthetic example. There are two good reasons for this extra step. First, it gives us a simple little sandbox where we can see what’s going on and sanity-check that the method behaves as we expect. Second, it lets us build an excellent benchmark that will help us interpret those fairly abstract standard deviations later on.

Extreme worlds: pure chance vs pure skill

Let’s start with two extreme scenarios. First, a game of pure chance, where the winner is literally decided by a coin toss. Second, a game of pure skill, where there is some fixed underlying skill ranking and the stronger player always beats the weaker one. What would the Elo distributions look like in those two imagined worlds?

In the totally random case, no player ever has any real advantage over another, so the “skill chips” just get tossed back and forth. Some winning streaks will occur, of course, but in the long run they’re balanced by losing streaks. Elo will keep nudging ratings back towards the middle, and everyone’s rating will hover near 0. The overall spread $\sigma$ settles into a very narrow band around 0.

In the opposite extreme, there is a fixed skill ranking, and the strongest player always beats everyone else. This top player will keep siphoning rating points from their opponents and never really settle at a final value. Elo is designed so that very large skill differences lead to only tiny rating changes, but in a world of perfect skill, there is always at least one opponent – the second-best player – who still gives them a little positive update every time they meet. The second-best player in turn keeps gaining points from everyone below them, and so on down the ladder. As a result, the strongest players drift further and further away from the pack, while the weakest ones sink lower and lower. In principle, the spread of ratings can grow without bound.

The p-deterministic game

With the extremes out of the way, we can now blend them into an intermediate case: the $p$-deterministic game. The idea is simple. We fix an underlying skill ranking for all players. Before each match, we flip a weighted coin: with probability $p \in [0,1]$ we play a game of pure skill, where that ranking decides the winner; with probability $1-p$ we play a game of pure chance, where the winner is chosen at random. This little Gedankenspiel is easy to understand and reason about. It gives us an idealised example of a game with “roughly $p$ parts skill and $1-p$ parts luck”, and it serves as the benchmark I promised — something we can later compare real games against. And because the rules are so simple, we can easily run simulations and calculate the resulting Elo distributions:

What simulations tell us

The first plot already shows the basic pattern: as we turn up $p$ and let skill matter more often, the Elo distribution gets wider and wider. To make this easier to see, we can just take the standard deviation $\sigma$ of each distribution and plot it against $p$:

The result is a smooth, monotone curve: higher $p$ consistently leads to a larger Elo spread $\sigma$. That gives us exactly what we wanted — a way to translate those abstract standard deviations into a more tangible “skill fraction” $p$. Later on, when we look at real games, we’ll be able to say “this game behaves roughly like a 70%-skill world” by matching its Elo spread to this benchmark curve.

What’s next

There’s still one big limitation left: everything so far has assumed two-player games. In the next part of this series, we’ll teach Elo to handle real multiplayer tables — the kind we actually have in modern board games — and only then move on to real-world data.

Interestingly, the Elo distribution for men’s tennis (ATP) looks more similar to the one for snooker than women’s tennis. ↩︎
Remember that $K=42$ I used in the snooker article? I promised I’ll explain in excruciating depth where it came from and I think I kept my promise. ↩︎

A brief introduction to Collaborative Filtering

contact@recommend.games (Recommend.Games) — Thu, 09 May 2024 21:47:57 +0300

What is a good recommendation?

Collaborative filtering is the workhorse powering the recommendations by Recommend.Games. Over the years, I’ve been asked every now and then how it works. So, I thought it’s high time I outlined the basic ideas behind our recommendation engine.

Let’s first take a step back and talk about recommendations in general. What is it we’re trying to achieve? The answer to this question is far from trivial, and it gets harder when you want to formalise its goals. Maybe a somewhat naïve approach would be to say that we want to recommend items that the user will like. But recommendations are as much about predicting what the user wants as what they didn’t even know they wanted. Sometimes the most “correct” answer is also the least useful: maybe our #1 recommendation is Wingspan and the user indeed would love to play it - but if they already knew about it, why recommend it in the first place?

To be honest, the solution that powers Recommend.Games pretty much ignores all those questions and asks a much simpler question: given the games a user has rated, can we predict how they would rate all the other games? We can then take the highest predicted ratings and use those to recommend games to the user.

The intuition behind collaborative filtering

Let’s make this a little more concrete and assume we only have six users we’ll call A through F. They’ve left the following ratings:

Game	A	B	C	D	E	F
Wingspan	10	?	3	7	?	?
CATAN	?	8	6	?	5	2
Gloomhaven	9	?	?	10	?	8
Twilight Struggle	?	7	5	?	6	?

Our task is to find a model to fill in the blanks. How would we go about this? The basic idea behind collaborative filtering is to find users with similar tastes and use their ratings to predict the missing ones. Let’s take a closer look at users C and E. They both seem to agree fairly well on their ratings for CATAN and Twilight Struggle, so it’s a fair guess that user E would rate Wingspan similar to user C. But we can also make this argument “in the other direction”: users seem to rate CATAN and Twilight Struggle in general similarly, so one game’s ratings should be a good predictor for the other’s. So since users F seems to dislike CATAN, it’s a fair assumption that they would also dislike Twilight Struggle.

Matrix factorisation via alternating least squares

That’s the high level idea behind collaborative filtering – but how do we actually implement it? If you know your Machine Learning 101, you might be familiar with linear regression, which tries to find a “line of best fit” for a set of given data points. We can think of collaborative filtering as linear regression in two directions: if we fix the games and try to predict the users’ ratings, we’re doing linear regression in the user space. If we fix the users and try to predict the games’ ratings, we’re doing linear regression in the game space. Collaborative filtering tries to do both at the same time – by alternating between the two, which is why this method is known as alternating least squares (ALS, “least squares” referring to minimising the squared error).

After running the algorithm, we end up with one vector for each user and one for each game. Taking the inner product (sometimes referred to as dot product) of those vectors yields a single number, the predicted rating. If we stack all the vectors for the users into a matrix $U$ and the vectors for the games into a matrix $G$, we can multiply those two matrices to get a matrix $R$ of predicted ratings:

\[ R = U^\top \cdot G. \]

In other words: we’ve taken the matrix of ratings from the table above and factored it into two matrices. So another way of thinking about collaborative filtering is as a matrix factorisation problem: we’re trying to find two matrices that, when multiplied together, approximate the original matrix as closely as possible.

Latent factors and embeddings

Those user and game vectors are also known as latent factors. That’s because they take the high-dimensional space of users and games and project it into a lower-dimensional space. While those latent dimensions don’t carry any particular human interpretable meaning, they capture the essence of what makes a user like a game. For example, one latent factor might capture how much a user likes games with a lot of player interaction, while another might capture how much they like games with a lot of strategic depth. Note that we can freely choose the number of latent factors, which is an important hyperparameter of the model. The more latent factors we use, the more expressive the model can be – but it also becomes more prone to overfitting. Recommend.Games uses 32 latent dimensions, which is a decent default choice for many recommendation problems.¹

The latent factors also have some interesting properties. For example, there’s a meaningful distance between them: if two users are close together in the latent space, they have similar tastes. The distance in question is the cosine similarity, which measures the angle between two vectors. Think of it this way: if the vectors for two users point in the same direction (cosine similarity close to 1), they have similar tastes. If they point in opposite directions (cosine similarity close to -1), they have opposite tastes. If they’re orthogonal (cosine similarity close to 0), they have no correlation.

The same goes for the game vectors: we can calculate the cosine similarity between two games to see if they appeal to the same users. Every game page on Recommend.Games has a “You might also like” section at the bottom. Those are the games with the highest cosine similarity to the game in question.

Vector representations with some measure of distance are called embeddings. So this is yet another way of thinking about collaborative filtering: we’re embedding users and games into a latent space where we can measure their similarity.

So, this was a pretty lengthy and technical article, but I hope it provides some intuition of how the recommendation engine behind Recommend.Games works. Obviously, there are a lot more details to it, but those are the most important ideas. One of the reasons why I wanted to talk about this topic is that we can have more fun with those latent factors, so stay tuned for more articles on this topic!

The number of latent dimensions also affects the size of the model. As mentioned, we have one latent vector for every user and every game, each with 32 entries. Currently, we have over 500 thousand users and over 90 thousand games, so the matrix $U$ has over $500\,000 \cdot 32 = 16\,000\,000$ entries and the matrix $G$ has over $90\,000 \cdot 32 = 2\,880\,000$ entries. Each entry is a 64-bit (8 bytes) floating point number, so the total size of the model is around $(16\,000\,000 + 2\,880\,000) \cdot 8\,\text{B} \approx 150\,\text{MB}$. It’s not a negligible size to load into the memory of a cheap server (which is what we’re using), but in the modern age of deep neural networks with gazillions of parameters, it feels almost tiny. ↩︎