Sam Wight

I help run a Mastodon server called Urbanists.Social. Established back in November 2022, it's now self-sustaining and gets donations double the amount it costs to host the server monthly. But it could be even more self-sustaining if it weren't for one inconvenient fact: self-hosting Mastodon is an absolute nightmare. Unless you're paying a wonderful company like Masto.Host (Hugo runs an amazing service), you're not going to have a fun time.

Hosting Mastodon yourself has a few unique challenges:

• You need to keep your users' data safe, secure, and encrypted.
• Your users expect to see posts from users on other servers almost instantly. That real-time nature is an important part of Mastodon.
• Your server is subject to extreme, unpredictable traffic spikes out of nowhere as our late capitalist hellscape causes more and more social media companies to make bad choices, forcing their users onto Fediverse servers that need to handle the traffic.

The second and third aspects of hosting Mastodon are challenging with how Mastodon is set up currently. Servers like mastodon.social struggle to scale even under moderate pressure with the large user base they have. Smaller instances are easily overwhelmed by a sudden burst of Fediverse traffic if they don't host Mastodon exactly right. And this is due to a few main reasons:

• Mastodon has a lot of moving parts that need to horizontally scale independently of each other.
• Mastodon is written in Ruby.

The last time I tried to set up a proper, self-hosted Mastodon setup was with K8S. It was an absolute nightmare to try and get done. This is a graph of what I remember the moving pieces looking like for Mastodon:

Every single one of these boxes needs to scale horizontally and / or vertically independent of the other one, and it's very, very hard to predict which one you're going to need to scale under high load. Each of them has different techniques and tricks and things you need to know in order to scale them and maintain them right. You need to back up your Postgres server but you don't need to back up Redis. You can get rid of Redis completely but you have to rebuild users' timelines somehow using a CLI command after you do so. S3 caches remote media and there's a job that will clear old media in it, but if you have a backed up Sidekiq queue it won't run.

In addition to this, Mastodon is just a lot less resource efficient than its peers. It's written in Ruby, and while Ruby is great, it's not known for it's incredible memory-efficiency and ease of scalability. Specifically in Rails, a lot of stuff gets pushed off to background jobs because responses need to be returned quickly (similar to NodeJS being single-threaded). There's also a lot of gems and the CLI commands can take a second to execute.

So usually what you end up with is instance maintainers who don't set things up in a way that makes it easy to horizontally scale each of these pieces, because that also requires much more maintenance. And so when Fediverse traffic increases, instance admins have to do a lot more work in order to keep up.

Fixing it

Fixing this is simple: don't use Mastodon. Use something like Pleroma or one of its forks instead.

Pleroma is written in Elixir. It eliminates a large amount of the items in the previous diagram:

• Elixir is built on top of OTP (Open Telecom Platform), which is incredible for scaling. Phoenix, the HTTP framework in Elixir, is roughly 10x faster and more resource efficient than Rails.
• Phoenix websockets are incredibly efficient and don't need a server like Redis. They just work out of the box.
• Background processing is done in Oban instead of Sidekiq. Oban uses Postgres instead of Redis to track jobs, and it runs on your application servers. It runs on the same servers that host requests. So you just scale up your application servers.
• Configuration management in Pleroma doesn't require full reboots. You can store the entire configuration in the database and edit it on the fly.

In the end you get a diagram that looks like this:

This eliminates more than half of the moving parts that need to horizontally scale. Instead, you only worry about two pieces: your application servers, and Postgres. Both of these can be automatically scaled and maintained. Managed Postgres DBs are incredibly cheap, and Elixir with cluster support set up properly will just scale up or down automatically as you add and remove more servers.

This is all assuming you don't have full-text search enabled on Mastodon, which a lot of servers do. Even then Pleroma just requires Postgres, as Postgres has full-text search. Sure it sucks a bit, but in my experience I've found it to be better than Mastodon's.

The only thing missing from Pleroma is some of the Mastodon API implementations. Trends and suggestions are unfortunately unimplemented. Some apps might appear broken with this unimplemented.

But anyways, I think that's the solution to fix Mastodon's scaling woes: not using Mastodon. Pleroma has amazing tools for moderation, including the Message Rewrite Facility (MRF). This allows you to write code that hides, deletes, or suspends accounts or posts immediately when they're received. It's stunningly simple.

That's all that. I'll be experimenting a little more with Pleroma (or probably Akkoma, one of it's better forks). I don't know of a migration path from Mastodon that's completely 100% seamless just yet, but we'll see.