We all know that a 4–1 finish at a major tournament is impressive. But is it harder to do at a larger event or a small local event? We analysed more than 22,000 Age of Sigmar tournament performances to try and find out.
Does Size Matter?
Imagine we have two Age of Sigmar players, and both have just finished GT events at the same weekend. Both are currently packing away their models after winning four games and losing only one, what the majority of us would consider a really good result.
But here’s the difference. Bob attended a small local event with 20 players, while Harry attended a major tournament with over 100 players. Which player do you think had the better result? I think most of us would immediately say that Harry had the better result. After all, walking away from an event with over 100 players having won four of your five games where the strongest players in the country are in attendance. So surely, going 4-1 at the larger event must be harder right?
But as we’ve been working on the statistics platform we’ve just published, we’ve now got enough data to start testing some of these assumptions rather than simply accepting them. We’ve got tens of thousands of match results, historical player ratings and information about the opponents each player faced.
So, for the first edition of Long Form Friday, I wanted to investigate: Is it actually harder to win four games at a bigger Warhammer tournament?
Why Tournament Size Might Matter
Before we dive into the data, we probably need to talk about Swiss pairings first. All the tournaments in our dataset have used Swiss pairings and consist of only five rounds. So events like LVO where you have brackets and 8 rounds of games haven’t been included in the data. Generally though, Swiss pairings will match players with another who has a similar result at that point in the tournament. So, win your first game and you’ll usually face another winner in round two, win again and you’ll probably encounter someone else who has won twice. The longer you keep winning, the more successful and experienced your opponents are likely to be.
But with Swiss pairings the number of undefeated players roughly halve after each round. So imagine a tournament with 32 players. As long as there are no draws, byes or other weird pairing complications, there would be 16 undefeated players after round one, eight after round two, four after round three, two after round four and just one after round five.
With a tournament of 64 players using the same assumptions, there could still be two undefeated players after five rounds and at a tournament with 128 players, there could be four. That’s because five rounds aren’t enough to guarantee a single undefeated player once the field exceeds 32 entrants, which is why we have events like LVO and similar how now have brackets and eight rounds of games over three days.
But this did give us a way to divide our tournament data. Instead of choosing random size categories, we could use the maths created by five rounds of Swiss pairings.
| Tournament Size | Theoretical Number of Undefeated Players after 5 Rounds |
|---|---|
| 1-32 Players | At most 1 |
| 33-64 Players | Up to 2 |
| 65+ Players | Perhaps 3 or more |
These are the perfect limits, not predictions of what will happen at every real tournament. Draws, byes etc will all throw something into the mix in terms of results.
But these are the categories that have create the main differences in results at a five round event. So this is where we’ll start.
What Data did we use?
For this, we used all of our recorded Age of Sigmar Fourth Edition tournament results. The dataset contains 923 recorded tournaments and 28,586 player performances.
For the main analysis, we filtered it to tournaments with only five rounds and only players who played in every round of that tournament.
This gave us:
- 864 tournaments
- 22,878 player runs
- 3,424 exact 4-1 finishes
- Roughly 4,400 finishes with 4+ wins
The sample covers tournaments of different sizes, from small local events to major national or international competitions.
| Tournament Size | Events | Player Runs |
|---|---|---|
| 1-32 Players | 602 | 10,236 |
| 33-64 Players | 214 | 8,593 |
| 65+ Players | 48 | 4,049 |
| Total | 864 | 22,878 |
How Many Players go 4-1?
We’ll start with an easy comparison. What percentage of players finish with exactly four wins and one loss?
Here we can see the affect of Swiss pairings, with roughly 15% of the players at any event going exactly 4–1.
Now let’s look at everyone who wins at least four of their five games. So this will include players that went 4–1-0, 4-0-1 and 5–0-0.
| Tournament Size | 4+ Wins |
|---|---|
| 1-32 Players | 19.47% |
| 33-64 Players | 19.19% |
| 65+ Players | 18.84% |
Again, thanks to the maths of Swiss pairings the difference is between the tournament sizes is small. With all events having roughly 19% of their players achieve 4+ wins.
But this is where we get to the meat of the matter because we’re treating every player as though they have the same chance of winning and, of course, they don’t.
WoElo
If you’ve followed Woehammer’s statistic, you’ll probably know about our WoElo system.
For anyone who hasn’t, WoElo is our player rating system. It uses historical tournament results to estimate the strength of players, which allows us to compare their performances and calculate chance of them winning each match. So for example, a player rated with 600 WoElo has a stronger historical record than somebody rated 350. But, it doesn’t mean the 600 rated player will always win. After all Warhammer has dice and plenty of chance for upsets.
But across lots of games, rating differences help us understand the chance of different outcomes, which gives us a useful way to look at tournament difficulty.
We can ask, does a player of a particular skill level have a lower chance of winning four games when they attend a larger tournament?
How Much Does Player Skill Matter?
We divided players into six WoElo brackets, based on their rating before the tournament.
These range from players rated 400 or below to those rated 700 and above. Then we worked out how often the players in each bracket achieved at least four wins.
| Pre-event WoElo | 1-32 Players | 33-64 Players | 65+ Players |
|---|---|---|---|
| 400 or less | 7.8% | 8.3% | 6.2% |
| 401-450 | 20.7% | 18.2% | 16.6% |
| 451-500 | 31.6% | 27.4% | 27.4% |
| 501-600 | 47.0% | 46.6% | 47.3% |
| 601-700 | 66.4% | 71.4% | 61.4% |
| 700+ | 82.8% | 83.7% | 83.6% |
Forget tournament size for a moment. Look at the difference between the lowest and highest WoElo brackets, players rated 400 or below achieve four wins in roughly 6–8% of their tournament appearances. Players rated above 700 do so in approximately 83%, which is a huge difference.
At the smallest events, the gap is roughly 75 percentage points. Then compare that with the difference between the tournament sizes, which is usually only a few percentage points a bracket. So player skill seems to be a better predictor of who can achieve four wins, and that might sound obvious, but it’s good to see just how large the gap is.
But are bigger tournaments harder for average players? Looking at the 401–450 WoElo bracket, we can see that at tournaments with 32 or fewer players, 20.7% of them achieve at least four wins. At tournaments with 65 or more players, that falls to 16.6%. We can see the same thing among the players rated 451–500, their success rate falls from 31.6% at smaller tournaments to 27.4% at larger ones.
But move up to the 501–600 bracket and the difference almost disappears, as at every tournament size, roughly 47% of these players achieve four or more wins. Above 700 WoElo, the success rate is almost identical across in all three sizes of tournaments.
So larger tournaments are more difficult for average players, but make no difference to elite players? Maybe.
But before we jump to any conclusions, we need to check something else. We need to check for other differences between the tournaments.
Controlling for Player Skill
The difficulty with just comparing percentages is that tournament populations aren’t the same all the time. Two players in the same WoElo bracket might still have different ratings. For example, a player rated 402 and another rated 448 both belong to the 401–450 category, but their expected performance isn’t going to be the same.
Plus depending on when the tournaments take place, they’ll be taking place under different Battlescrolls, and the countries of the events may also change the competitive population.
So we used a statistical model to take this into account. We modelled the chance of achieving at least four wins using:
- Pre-tournament WoElo, allowing for a nonlinear relationship with success.
- Tournament size, using our three Swiss-based categories.
- Battlescroll.
- Country.
The result gives us an adjusted comparison, so what happens to the estimated success rate when we compare tournament sizes while standardising the player-rating and other recorded characteristics?
The difference between the smallest and largest tournaments is very small at only 1.34 percentage points.
Our model estimates that players attending events with 65 or more players have a slightly lower chance of achieving four wins than similar strength players attending events with 32 or fewer players.
The estimated odds ratio is 0.895, with a 95% confidence interval of 0.811–0.988. The associated p-value is 0.028.
In statistical terms, it provides evidence of a difference, though it’s not a large one. The medium sized tournaments, aren’t statistically different from the smallest events in this.
So we can start to answer our question. Yes, larger tournaments appear to be slightly harder when we also take into account player skill. But the difference is smaller than I think most of us would have thought.
Is this based on only a few major events?
Whenever we’re looking at tournament data, there’s always a problem with the biggest events. Just one major tournament can give us hundreds of player runs, which means that an unusually strong or weak event could affect the results.
So we repeated the adjusted analysis, gradually removing the largest tournaments. First, we removed the largest event, and then the five largest, until finally we removed the ten largest.
If the disadvantage at bigger tournaments was down to one or two strange event results, we’d would expect the result to change a lot, but it didn’t.
The results stayed consistent, and across all four models, the estimated disadvantage with the largest tournament category stays between 1.2 and 1.4 percentage points.
This gives us more confidence that the what we’re seeing isn’t because of one strange major tournament. But it doesn’t mean tournament size is the main reason. This is still based only on what we’ve seen, and there may be differences in future events that we haven’t modelled for.
What about the strength of the tournament players?
The other reason larger tournaments might be harder, is that they could attract stronger players. We can look into this by checking the average pre-tournament WoElo rating of players across our three tournament categories.
| Tournament Size | Average Pre-Tournament WoElo |
|---|---|
| 1-32 Players | 393.8 |
| 33-64 Players | 393.5 |
| 65+ Players | 407.8 |
There’s two things here, firstly there’s very little difference between the smallest and medium sized tournaments, and both have an average player rating of roughly 394.
Second, the largest tournaments have a stronger average player base, at roughly 408. So the largest tournaments in our sample do attract stronger players on average and that may help explain why they’re slightly harder for a player.
But field strength and tournament size aren’t the same thing. A small event packed with experienced competitors could be stronger than a larger tournament attended by a more varied group.
If we really want to understand a tournaments difficulty, we need to look at who attended, not just how many turned up.
How Impressive is a 4–1 at a Major?
Let’s go back to Bob and Harry, our two players we talked about at the beginning of this all. Bob finished 4–1, one went to a 20 player event, and Harry went to a 100 player tournament.
Which is the more impressive result? Based on what we’ve seen, I wouldn’t want to judge them on the tournament size on its own. The larger event may well have been harder, and our analysis does seem to suggest that there’s a small disadvantage with larger tournaments, even after taking into account player skill, but the difference is very small.
And the skill of the opponents matters a lot more. A 4–1 finish at a small event against four high rated opponents could be much more difficult than the same record against weaker opponents at a larger event.
Also, an inexperienced player who goes 4–1 might have performed well above expectation than an elite player who did the same.
If we want to compare tournament performances fairly, we need to take into account player’s skill and the skill of the opponents they faced and the results they were expected to achieve.