Why The Algorithm Hates You

And other adventures with sampling bias

Please hit the ❤️ “Like” button if you enjoy this post, it helps others find this article.

Most of your friends are more popular than you. Most people you see at the gym are fitter than you. The lane of traffic you're in during a traffic jam is usually slower than the other lane.

Now let me hit my fellow Substack writers where it hurts: The Algorithm really does favor the Notes you see more than yours. Most of the writing you see on Substack is better than yours.

No, this isn't me trying to affirm your depressive thoughts. This is literally just math—these statements are going to be true for most people (assuming you do things like writing, going to the gym, or having friends).

The Friendship Paradox

If someone has very few friends, you're not likely to be one of them. If someone has a lot of friends, you have a better chance of being one of them.

Consider this friendship network:

Source

A line between two people indicates they are friends. Beside each person is a list of all of the friends their friends have.

Literally everyone is friends with Bob. That means Bob appears on everyone's list. But since Bob is friends with everyone, he has more friends than anyone else. So everyone (other than Bob) is friends with someone with more friends than they have. Poor Dave is only friends with Bob, so he is comparing his one friendship to Bob, who has three.

Alice, Carol, and Dave all have fewer friends than the average number of friends their friends have.

Obviously Bob isn't less popular than his friends, but he is the only one.

The above is obviously a made-up network. But this dynamic of people having friends who on average have more friends than they do shows up with real friendships as well. Consider this "sociogram" of a 1st grade class:

Source

Each node is a child in the class, and the lines between them indicate who a child wants to sit next to—in other words, who they consider a friend. Some kids are more popular than others (CE must be a really cool kid). But notice how all of those kids surrounding CE have very few lines connecting them to the rest of the class. Everyone who is friends with CE is less popular than CE. And there are a lot of people who are friends with CE!

If you take the average number of friends of each kid, for most kids it will be less than the average of the kids they are friends with.

This is called the friendship paradox, but it isn't really a paradox, and it isn't a feature only of friendships. It's a mathematical principle. Unless everyone has the same number of connections, you're going to run into this.

The underlying principle is sampling bias. The very thing we are measuring (number of friends) impacts how often we sample it (how often someone shows up as a friend)—like how Bob was on everyone's list above. Let's take this general principle to the other examples I listed in the introduction.

Variations on the Friendship Paradox

Consider the gym example. Whenever you go to the gym, those that spend more time at the gym are disproportionately likely to be there. The gym bros that spend hours at the gym everyday are more likely to be there than the person who goes once a week for 30 minutes (if they're feeling up to it).

If you peek into the gym at any given time, people who are there more than average are going to be overrepresented.

Given that those that spend a lot of time at the gym are more likely to be fit, we have the same principle: the thing we are measuring (fitness) is related to how often we sample it (how often someone is in the gym).

What about traffic jams? The lane packed with more cars is going to move more slowly. On average, you're going to be in the lane with more cars, since there are more people there. The experience of being a person in the overpacked lane is overrepresented.

Why The Algorithm Hates You

Substackers, do you see where this is going yet?

(For those of you who are not Substackers, in what follows replace "Notes" with "posts" and think along the lines of Bluesky or X/Twitter or anything else that algorithmically places posts in a feed.)

The reason you see a bunch of crappy Notes that get a million Likes isn't because everyone else's crappy Notes tend to get Likes. It's because the Algorithm puts things in your feed based (at least partially) on how popular they are.

You don't see all the Notes that don't get a lot of Likes. You are disproportionately being shown the popular stuff. Occasionally, an otherwise unnoteworthy Note will break through and get a lot of engagement for whatever reason, and that's the one you'll see.

The Notes that end up in your feed are going to be more popular than you would expect given their content. The Algorithm that serves up your feed is trying to optimize for something. Some Notes are going to get lucky—maybe the first few people the Algorithm showed a Note to happened to be in the mood to Like pretty much anything. That Note is going to be more likely to show up to a wider set of people because of the engagement it drove, and it will show up with those lucky Likes. That means lucky Notes are going to be disproportionately represented in your feed.

Again, the very thing we are counting (Likes, or some other measure of engagement) is related to how often we sample it (it showing up in Feeds).

Not that it's all luck—the Feed algorithm is trying to give you things that will drive specific engagement from you, and the popularity of a Note is one feature it's looking at. But to the extent The Algorithm is using popularity to gauge what to put in your Feed, you'll on average see things that are disproportionately lucky in your Feed.

Why You Feel Like Your Writing Sucks

Notes aside, why does it seem like all the writing you come across is better than yours? Are you really a worthless waste of space who will never amount to anything and should give up on all of their dreams and ambitions? Maybe. But putting that aside, the reason you come across writing that's better than yours is two-fold, one part similar to the Notes example and one similar to the gym example.

First, good writing is going to be more popular, and therefore served up more in your feed or any other method of discovery more often. If people like something, it's easier to find. So again, the thing we are counting (the quality of the writing) affects how often we sample it (how often it shows up in feeds or people sharing it).

Second, people who write a lot create a disproportionate amount of the content, and people who write a lot are on average going to be better writers through practice. The average writing is written by a writer who is better at writing than the average writer, because above-average writers write more writing.

Sampling Bias Affects How We See The World

So cut yourself (and The Algorithm) some slack. Because of sampling bias, you'll always get a distorted view of the world, and that distortion is often in ways that would make you feel bad about yourself.

I've written previously about how sampling bias distorts our views of success and contributes to many of our cognitive biases. Sampling bias plays a pretty big role in how we see the world, since it literally shapes what parts of the world we see.

Things that are dramatic are going to be overrepresented on the news, giving you the idea that the world is more dramatic than it actually is. But again, the thing we're measuring (how dramatic something is) impacts how likely we are going to see it (it showing up on the news, or someone sharing it on social media). Unfortunately, given people's negativity bias, "dramatic" generally means "negative", so we disproportionately are fed negative news.

That something is noteworthy and dramatic enough to make its way to you means it's not business-as-usual. We don't get news reports about all the people living happy normal lives or all the places that are not suffering natural disasters. Even before considering various other reporting biases that might occur, there's good reason to hold certain types of news reporting at arms-length if you're trying to get an accurate picture of the world. Otherwise you might get a rather inaccurate, pessimistic view of things.

Please hit the ❤️ “Like” button below if you enjoyed this post, it helps others find this article.

If you’re a Substack writer and have been enjoying Cognitive Wonderland, consider adding it to your recommendations. I really appreciate the support.