Personally I proxy everything through cloudflare tunnels using a wildcard cert, and have Authelia in front of everything. No open ports.
Haven’t had bot problems since swapping to this setup.
I found that using a wildcard domain cert for HTTPS and an uncommon higher port number works just fine to avoid crawlers and scrapers. I also hide my ssh port behind WireGuard because I got annoyed with the bots filling up my logs with failed attempts. I’ve been doing that for nearly a year now and I have had no more issues with bots
I’m going off the assumption that the people who run crawlers and scrapers are going for known common ports for quick ease of access
I don’t feel the need to run anything extra such as Anubis either. The inconvenience of using an uncommon higher port number has been minimal and I still get to enjoy my tiny corner of the internet
I have a simple static site. It’s been up for less than a month. On the cheapest possible VPS, someone gave me. Robots.txt:
User-Agent: * Allow: /How long do yall think it will take for it to blow up?
If I didn’t enjoy the cat & mouse of it, I’d definitely walk away from running a site. Just out of curiosity, I created an hourly graph today showing malicious bot, legitimate bot and human ratio graphs for one of my sites. The malicious hits make up around 80% hourly and human hits are usually well under 10%.

I host a few sites and I haven’t had a problem yet. Sure, logs suggest a majority of traffic is bots, but the cheap VPS I’m hosting on hasn’t melted so I haven’t been motivated to do anything about it.
Lemmy is particularly susceptible. Hosting a static site with a limited number of pages is fine. But Lemmy needs to check all posts, find the votes, sort them, and compile the page. It does that for practically every page load, and so loading a page once is quite “expensive”.
Then add on to that the almost unlimited pages of content for bots to crawl through, bots just get caught endlessly scraping pages. Even powerful servers struggle, which is why most Lemmy instances have implemented some sort of protection against bots.
I believe you, but the linked article is not about hosting Lemmy.





