So, I joined the Fediverse.
Why I Left The Bird Site Now
Long-time readers of this moribund blog would know that I have had a Twitter widget on my sidebar for the good chunk of a decade. The widget will probably disappear eventually, along with the website that populates content from it. The reason for this is very simple. Twitter is now owned by a billionaire that overpaid to buy the platform for the memes, and is desperate to recoup his investment by making absolutely idiotic moves to destroy the platform as quickly as possible.
At least his accountant will have some losses to write off next year.
The reality of it is that I actually haven't been using Twitter "as intended" since 2016 at the least. I locked my primary account and used a branded Fantranslation.org account for blog post announcements. The only people I bothered following or interacting with were a handful of software developers, and I deliberately self-censored my opinions to avoid getting in the Twitter limelight. Trust me, you don't want to be in the Twitter limelight for the same reason you don't want to be a celebrity on 4chan.
Of course, this isn't uniquely a Twitter problem, but I've found Twitter to be the line that I drew in the sand. If your platform is more toxic than Twitter, than I won't touch it, because I can at least manage Twitter's toxicity. With Elon Musk's hands on the wheel I have no confidence that will continue to be the case.
...And Not Any Earlier
I am a decentralized social media skeptic.
Social media platforms may be legally shaped like content hosts, but they act more like broadcasters. One of the oldest problems on the Internet has been spam control; you can circumvent free speech by shouting over people if that speech is also gratis. The only reason why the Internet - as in, everything from e-mail to the World Wide Web - even works is because of a lot of really underappreciated people working to find and remove spam.
Decentralization explicitly calls for getting rid of the capability to remove content in the name of free speech. This causes problems. If you remember, say, Gnutella (you dirty pirate); you probably also remember a point in time where literally every search gave you back a bunch of malware downloads and spam. This is possible because gratis speech - or, "free as in beer" speech - allows people to talk over you. The way antispam works is to find some way to make spam expensive so that spammers go away.
This also applies to toxicity and harassment. In fact, the most common way to harass someone online is to throw lots of spam their way, because it tends to take down servers and make using the platform difficult. This even has its own name: a DDoS attack.
Centralized social media isn't particularly better than decentralized at preventing this; but it at least provides chokepoints to shut down bad actors. Having a bigger server behind it also means it's harder to DDoS. This is something I am giving up moving to the Fediverse.
When I first posted on my Mastodon account, I said to consider my presence there to be "signed 'under protest'". Fortunately, it seems the Mastodon developers already have the same concerns that I do. Virality is considered an anti-feature; scraping the Fediverse is a sin; and lots of old-timers on the platform are actively warning people not to expect a Twitter-like experience.
Let's see how long that lasts.
How Mastodon Works, And How To Get On The Fediverse
So, Mastodon can be used in one of two ways:
- You can sign up for an account on a public instance, if you don't want to fuss with server administration. This is like just signing up for a regular social media account.
- You can host a private instance for yourself, for the fully semi-sovereign decentralized experience. This is like running your own WordPress blog.
Uniquely for a social media platform, people on different instances can interact and discover one another as if they were on the same one. This is thanks to the power of federation - when you interact with another server's content, your instance uses WebFinger to authenticate users by domain name and ActivityPub to send cross-instance notifications.
The sum total of all instances that implement these standards is called the Fediverse.
Since I wanted an @fantranslation.org handle, and I already self-host this blog; I decided to self-host Mastodon as well. This requires jumping through some hoops to redirect WebFinger through my Drupal install. I could have alternatively found or wrote a plugin for Drupal that makes it speak WebFinger and ActivityPub natively, which would let the blog show up on and be followable from Mastodon instances. Yes, the Fediverse is not just cross-instance, it is also cross-platform. However, I would probably have to make significant changes to the structure of the content to make this still look like a normal blog and not, say, put every reply I make on the front page.
The Pitfalls of Self-Hosting
Fantranslation.org is already hosted on Amazon Web Services, so setting up Mastodon is simply a matter of following the self-hosting guide they provide. In addition, I also would like to impose a challenge upon myself: can I do this for less than the cost of a Twitter Blue checkmark ($8/mo)?
In theory, yes!
In practice, I hit a number of roadblocks and traps for the unwary:
- Mastodon locally caches all media it encounters to avoid giving other private instances the hug of death. This is perfectly fine for large public instances, but the caching mechanism provides less value if you're the only one on the server. Currently, my instance has at least 7.68GB of other people's media on it and growing.
- All that media is expensive to store on EBS volumes. S3 storage is 80% cheaper and can be made even more cheap through Intelligent Tiering (though I am not sure if that is worth the per-object automation charge). If you are hosting on a cloud server S3-compatible storage is not optional.
- Mastodon uses a lot of RAM. Or at least that's relative to this Drupal blog which could fit comfortably in a t2.nano with 512MB of RAM. I've expanded it to t3a.small which is 2GB of RAM and blows past our $8/mo budget with an On-Demand price of $13.54/mo. (t4g instances are cheaper but require a full OS rebuild to ARM64.)
- Don't be tempted to rely on swap space to fit in a smaller instance size. This is because all "general purpose" EBS volumes have an I/O burst balance relative to their storage size. Provisioned IOPS is even more pricey than regular EBS SSDs. If your instance starts using the swap space for active-set data, it will grind to a halt when you least expect it.
- Amazon S3 recommends you disable Object ACLs. Don't. If those are disabled, all S3 requests that try to set an ACL will fail, including and especially uploads. Mastodon does not have a setting to allow uploads to S3 without an ACL; since it wants to be able to hide objects from banned users. (Yes, you can get banned on Mastodon!) This is not well documented.
- The last few items caused a cascading failure on my server. Specifically, Mastodon will retry jobs that fail due to a misconfigured S3 bucket. Since I had swap and the media upload job did not fit into 1GB of RAM, it was constantly swapping in and out, which ate up my EBS burst balance until the server became unresponsive.
- If you want working e-mail from your instance (say for password resets) you have to sign up for a third-party e-mail delivery service. Amazon SES requires pre-approval to send to unverified inboxes (you can just verify yourself if you're self-hosting and stay in sandbox mode). EC2 also bans outbound port 25 without an even more onerous pre-approval process.
- I made the mistake of setting up a VPC Endpoint when setting up Amazon SES because I didn't know about the port block. (As for why an Amazon service is port-blocked by Amazon, beats me.) It worked, but it's expensive - endpoints bill a penny per hour, per availability zone, which is about $21.60/mo. Just use port 465 with SSL and the public SES endpoint.
- No one tells you this, but put everything behind Amazon CloudFront if you can - your EC2 instance, your S3 bucket, everything. It has a 1TB/mo permanent free tier.
- Set up snapshot backups for your root volume, just in case.
- I also had to consolidate a bunch of old servers and services into my t3a.small instance to save space. Notably, this Drupal blog's database used to be hosted on Amazon RDS, which is absolute overkill and cost more than the t2.nano it used to run on. I didn't notice since this was only costing around $20/mo in total.
All of these hurdles were surmountable, of course. But our self-imposed "cheaper than a blue checkmark" budget is blown just by the application server on EC2.
It's More Money, Unless We Are Willing To Cheat
Or, it would be, except for my secret weapon: Reserved Instances.
So far, we've been only talking about the On Demand pricing of EC2. This is where you just say "I want a web server" and Amazon rents one to you. There are two other pricing models Amazon supports:
- Spot Pricing, where you make a bid for a particular amount of compute and the instance only runs when the spot price for that instance is met. You can save about 60-70% this way but with the risk of interruption if a bunch of new instances spin up in your availability zone.
- Reserved Instances, where you commit to renting a certain instance from Amazon for an amount of time. You can also use Savings Plans for a more flexible approach. These are about 50% cheaper than On-Demand for a 3-year plan.
I would not recommend spot pricing for a persistent server like a blog, unless you're also paying for an Elastic Load Balancer and have non-spot instances to take up the slack. However, reservations and savings plans can still save money if you're willing to commit to a particular instance type or total spend. So, I may be able to meet the budget I set, with the downside of not being able to cancel and shut this whole thing down if I decided I don't want a website at all anymore.
Incidentally, the part where spot pricing really shines is for intermittent compute like CI workloads. If you remember, Paparouna CI used to use EC2LatentWorkers for a per-build cost of like pennies on really big instance types that would be prohibitively expensive to rent continuously. Unfortunately we stopped doing this because it broke and I haven't been arsed to fix it.
So How Does It Feel, Punk?
After a week of playing DevOps maestro, I feel like the new instance is at a steady state and functional. Furthermore, I'm having more fun using it than I am using Twitter. This is something I didn't expect, honestly - given my reservations above. I'm actually writing more on there and talking more about my various projects.
The reason for this is because of a number of "anti-viral" features inherent to federation. Namely, if I post something stupid, it doesn't immediately hit all of Twitter. It has to crawl through servers whose users follow me and ActivityPub relays that I've explicitly applied for. This is like how YouTube has an algorithm that tries to shuttle you into narrow interest categories, except it's entirely manually operated.
Want to follow Art Fediverse? Just find people posting art and follow them. Want to follow emulator developers? Find them and follow them. Want to get noticed in those circles? Just use the hashtags corresponding to what you're putting out. There is no algorithm and our timelines are entirely chronological. No "here's posts from two weeks ago, ICYMI" or "someone paid way too much money to boost their Tweet" nonsense. It's just things you want.
To be fair, I still don't think it'll last, but I'm going to enjoy it while it lasts.