If I have several backends that more or less depend on each other anyway (for example: Lemmy + pict-rs), then I will create separate databases for them within a single postgres - reason being, if something bad happens to the database for one of them, then it affects the other one as well anyway, so there isn’t much to gain from isolating the databases.
Conversely, for completely unrelated services, I will always set up separate postgres instances, for full isolation.
Nice post, I enjoyed the storytelling. Glad it’s all sorted now 😁
Btw, regarding this point:
All in all, this has been a fairly frustrating experience and I can’t imagine anyone who’s not doing IT Infrastructure as their day job being able to solve this. As helpful as the other lemmy admins were, they were relying a lot on me knowing my shit around Linux, networking, docker and postgresql at the same time. I had to do extended DB analysis, fork repositories, compile docker containers from scratch and deploy them ad-hoc etc. Someone who just wants to host a lemmy server would give up way earlier than this.
I think you’re totally right, but at the same time, I think the collaborative troubleshooting that happened on Matrix (and has happened many times in the past for other issues) is pretty healthy, and not something that is always possible for other open source software.
They specifically called it “child abuse content”, not “child abuse”. This seems perfectly valid, no?
By the way, just because these are digital renderings does not mean that there is no harm. Seeing such content can still be harmful to past victims. Just try to put yourself in this situation: imagine just playing some video game online, and suddenly being exposed to people recreating traumatic experiences from your past. Not only that - you also discover that the creators of the video game are involved & actively enabling such content. Seems completely messed up to me.
That particular instance was very recently the source of a lot of CSAM and spam, so that’d be why. A lot of instances recently upped their security to combat that.
Just to add some more context, there was an attacker recently who created accounts on several Lemmy instances and used those accounts to spread CSAM. On lemm.ee, this attacker created 4 accounts over a 24h period, but was not able to upload any CSAM to our servers due to our stricter upload rules (we require 4 week old accounts to upload any images at all), and all of the 4 accounts were removed very shortly after creation (most of them within an hour of signing up). The attacker gave up trying to use lemm.ee very quickly, and moved on to other instances.
I just wanted to share this context to illustrate that while indeed the different measures we implement to protect the instance can have a negative impact on legitimate users, I really believe that overall, they have a net positive effect. In addition to Cloudflare DDoS protection and image upload restrictions, we also have a separate content-based alerting layer on top of Lemmy, which allows our admins to quickly notice when something suspicious is going on. As another example, this alerting has allowed us to extremely efficiently deal with a current ongoing spam attack on the Fediverse, and I bet many lemm.ee users aren’t even aware of this attack due to the quick content removal. We will continue to improve our defenses, and hopefully try to limit the impact on real users as much as possible, but some trade-offs are necessary here in order to protect the overall userbase.
The nice thing about Lemmy is that you can always host your own instance, even if it’s only for your own individual use. You can basically use your own instance as a proxy - other instances will not see how or from where you are connecting to your instance.
Large instances are being attacked almost constantly at this point in smaller and bigger ways. Almost all measures we implement to combat these attacks come with some trade-offs for the rest of the userbase.
Good luck with the update! One great thing about 0.19 is that it allows users to check federation status between instances, will be awesome to get that for lemmy.world as well.
What exactly is the issue with our admins? If you feel you’ve received some unjustified moderation, feel free to contact me and I can have a look.
I’m sorry for your experience and I hope you’re doing better now. It’s very important to note that the major victims in this “attack” are absolutely the abused children. That’s exactly what makes this the most disgusting attack on Lemmy to date.
Cloudflare CSAM protection is not available outside of the US, unfortunately.
As a test, I ran this on a very early backup of lemm.ee images from when we had very little federation and very little uploads, and unfortunately it is finding a whole bunch of false positives. Just some examples it flagged as CSAM:
Do you think the parameters of the script should be tuned? I’m happy to test it further on my backup, as I am reasonably certain that it doesn’t contain any actual CSAM
Any thoughts about using this as a middleware between nginx and Lemmy for all image uploads?
Edit: I guess that wouldn’t work for external images - unless it also ran for all outgoing requests from pict-rs… I think the easiest way to integrate this with pict-rs would be through some upstream changes that would allow pict-rs itself to call this code on every image.
This approach makes so much sense from a business perspective.
How many here have this experience: out of my entire friend group that I grew up playing video games with, I can’t think of a single person who kept pirating games after acquiring disposable income, even though we all exclusively played pirated games as teenagers. Without piracy, none of us would have had access to any games, and very likely none of us would still be into gaming today, spending probably thousands of euros every year on games, consoles, PC components, etc.
Hey, I saw this ping, but I didn’t actually get any message from you about CORS headers. Where did you contact me?