Mastodon Is DDoSing Me
Whenever I, or someone else, posts a link to this blog on Mastodon, it DDoS's me and brings the site down for a couple minutes.
Over the last few months, I've noticed that whenever I (or someone else) posts a link to this blog on Mastodon, the decentralised nature of the platforms effectively DDoS's me.
A DDoS is a Distributed Denial of Service, where multiple remote servers all send traffic at the same time, which overwhelms the target. The "target", in this case, is my server. 😢
Why is this happening?
So when you post a link to an external source to pretty much any social media site, the social site pulls some meta data from the remote site and creates a little card that makes the link look pretty. For this site, it looks something like this:
The social site attempts to pull things like the feature image, post title, description and blog name. If the link is posted to a centralised service, like Facebook or Twitter, it's not a problem as there's only one source requesting that meta data.
But Mastodon is different.
As many of us know, Mastodon is decentralised, which means that the network is spread among multiple servers, or instances. So when a link is posted, every instance where you have a follower requests this meta data independently.
Problem is, I have around 26,000 followers on Mastodon. According to FediDB, there are around 27,000 instances on the Fediverse.
So let's say, conservatively, my 26k followers are spread across 1/4 of the instances that are out there. That's still 6,750 servers that request that meta data, every time I post a link to this site, at the same time.
The result is a DDoS that takes down my site for a couple minutes.
Brilliant.
What can I do about it?
In short, not much. I've tried serving the feature image from a CDN to take some of the load off my server, but that hasn't worked.
I'm considering putting this site behind Cloudflare to see if that helps, but that makes me feel really icky. So I'd only do that as a last resort.
What can Mastodon do about it?
This problem is on them to fix, really. It's a bad look that as users become more popular, they're effectively DDoS'd. At this point, I'm very reluctant to post links direct to people's blogs, as it will likely bring their site down. Instead, I create link posts so this site feels that pain, not the site owners.
The Mastodon team have apparently implemented a temporary solution where instances will wait a random time between 0 and 60 seconds before they fetch the meta data, but I'm yet to see that work for me.
They're also working on a long-term solution, apparently. Renaud Chaput, the Mastodon CTO was quoted in this post on The Register saying:
We have a mitigation in place as servers are waiting a random time between zero and 60 seconds before generating the preview to avoid sending all the requests at the same time, but a proper fix would be to have the link preview information shared between servers (federated) so each server does not need to fetch it. We have several ideas on how this could work, but we also need to ensure that this will not cause other issues, like allowing those to be spoofed.
We do not consider this as a critical issue because you need accounts on thousand of servers to follow an account for this to generate a non-trivial amount of requests, especially now that they are spread over 60 seconds, and there are much easier ways available to achieve the same result than using the Fediverse.
Well that's bullshit. I'm far from the largest account on Mastodon. Yes, I have a lot of followers, but I'm not really an anomaly. Mastodon is effectively DDoSing lots of sites across the internet, if that's not a critical issue, I don't know what is.
Drew DeVault said this in a GitHub issue on the topic:
It is the responsibility of software like Mastodon to be a good neighbor on the internet. DDoSing others is not being a good neighbor! It's important to figure out how to prevent this issue from occurring.
Couldn't have said it better myself, thanks Drew.
I've thought about the scalability of Mastodon a number of times. As the co-admin of a fairly large instance, and knowing what it costs to run that instance, scalability on Mastodon is an issue, I think.
These kind of issues are a fundamental challenge of any decentralised network. And I totally get that they're not a simple problem to fix. But the fact is, for something to be decentralised, scalability is a challenge. I don't know how the Mastodon team plan to fix these issues, but if the network is going to continue to grow, they need to come up with some solutions.
Please, Mastodon, stop DDoSing your users.
I wrote an update on how I worked around this issue. You can read about that here.
Want more?
So you've read this post and you're still not satisfied? Ok then, here's some other stuff for you to do: