Updated my Mastodon instance o 3.0.

@angristan @benoit btw, right now all image attachment requests to return 503:

Maximum number of server active requests exceeded



You should use backblaze behind CF (free bandwidth) and not wasabi.

@benoit @codewiz does backblaze use S3 now or do they still have their own thing?

Hmm good question. I think so. At least you can add minio as a gateway.

@benoit @angristan @angristan @benoit I'm far behind on all the cool cloud services that you kids use, but...

...I worked on the the serving infrastructure of, which includes all of the thumbnails of YouTube and Image Search, as well as fonts and petabytes of other highly cacheable content.

Generic storage systems and generic web servers are a poor solution for serving small static files efficiently.

I'd be happy to help come up with a scalable design for Mastodon.

@benoit @angristan The very first thing you should ask when picking the storage system is: what's the tail latency for reading small objects at high qps?

Caching can reduce average latency but can't do much with tail latency. You should care because pages can't fully render until the slowest font or JavaScript file is done loading.

@benoit @angristan The second question is: does the datastore continue to perform well when the write qps increases? Some blob stores must do periodic garbage collection or replication which impacts latency.

If your datastore supports replication, can you create serving replicas in americas, europe and asia? Can you tie http frontends to those same zones? Can you redirect user traffic to the closest replica? Can you spill traffic to the other replicas when the closest one is overloaded?

@benoit @angristan Caching: you want large amounts of ram, possibly shared between the http frontends, but located very close to them (<1ms). Something like memcached, I guess.

@benoit @angristan The webserver for static files should be something statically compiled. Avoid anything with GC unless you know how to avoid GC pauses on the serving path.

Serving static files is starting to approach a realtime system, although deadlines are not that hard. As long as your 99-percentile latency is consistently low, your service will feel snappy to most users, most of the time.

@codewiz @benoit can these scaling techniques be applied to Mastodon though? Instances should probably not come close to this scale, ever?

@codewiz @benoit even for, a simple bucket with a reverse proxy and CDN caching in front will work perfectly. I don't see the point of going super hardcore on this stack if the the storage provider is down most of the time. Moving elsewhere seems much more efficient

@angristan @benoit Don't instances also need to serve images originating from other instances? As the network scales in size, storage costs will grow for all instances.

In centralized platforms, the cost per user goes down as the number of users grows... classic scale economy.

I'm still not convinced that the current fediverse architecture can scale up to 1B users by just adding more instances, unless they find a way to share some of the serving infrastructure.

@angristan @benoit If each instance buys separate Amazon S3 buckets to store images and serves them via Cloudflare on separate domains... we're effectively as centralized as we could possibly get, while having low cache-hit rate and huge storage costs per instance. I couldn't possible imagine a worst serving architecture for Mastodon 😦

@Gargron, what do you think? Is there a hope for instances to share some content serving infra in the future?

@codewiz @benoit @Gargron

> share some content serving infra in the future

The issues with this are:

- trust (integrity etc)
- who will serve, so who will pay
- who's the source of truth

@angristan @benoit @Gargron Content-addressed systems can solve the first and the last problem elegantly.

Similar to git, you could clone a repository from an untrusted source if someone you trust gives you the hash of the revision you wanted.

Problem is, the client should verify the content before using it, and perhaps computing hashes from JavaScript is too inefficient.

@codewiz @benoit yes, I absolutely agree that the fediverse isn't ready at all...

@codewiz @benoit nerds are everywhere saying that we should decentralize everything and that centralization is the devil but heh, centralization works, and scales, at least

@codewiz @benoit I mean decentralising by deduplicating everything seems very inefficient

@angristan @benoit Our intution of how distributed systems work is often wrong. Amazon S3 is centralized from the administrative pov, but it's highly distributed and redundant in the ways that matter for scalability and high availability.

The fedivers is administratively distributed, but each instance is a SPOF from the point of view of the users on that instance.

Your instance suddenly went down and you didn't dump all your data the day before? Sorry, you lost all your toots and your network!

Agreed, overkill, isn't even close to 1% of gstatic. And for mono instance like mine, nginx+ext4 does well, no need for any fine tuning :p.

@benoit @angristan Presumably your instance only holds the images you post and those from the users you follow. Can you check and tell me how many objects you have now? And what's the daily growth?

The single-user instance is an interesting case, because it gives us a baseline to characterize how storage costs vary with the number of users per instance.

@angristan @benoit Very interesting data, thank you.

Hint: you should graph it... for science! 🙂

@angristan @benoit I want to know: if you ran a mapreduce over the entire dataset and hashed everything, how many duplicates would you find? And how many images are effectively orphans (uploaded but never attached to a toot, or attached to a toot that was deleted)?

@codewiz @benoit Gargron told me that there is a sidekiq job that deletes images in that second case (uploaded but never attached)

@codewiz @benoit Yes I really have to move away from Wasabi. The instance has become unusable in the last weeks.

@angristan @codewiz @benoit I'm wondering, what's the storage, average bandwidth and peak bandwidth use you're seeing? Is running storage on one or a few small VPSes viable?

@angristan We know you are very busy these days... No pressure. Maybe put a service announcement somewhere so that users know you're already aware of the issue.

Sign in to participate in the conversation

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!