Send CathInfo's owner Matthew a gift from his Amazon wish list:
https://www.amazon.com/hz/wishlist/ls/25M2B8RERL1UO

« previous next »
Pages: 1 [2] 3

Author Topic: CathInfo swarm of bot traffic  (Read 417 times)

0 Members and 1 Guest are viewing this topic.

Offline Matthew

  • Mod
Re: CathInfo swarm of bot traffic
« Reply #5 on: February 19, 2026, 07:55:51 AM »
In response to TraditionalSermons --

I didn't mention all the work I did with the server logs. I have the ability to look at an aggregate every day, showing the top 366 (yes, that exact number) IP addresses that hit the site. I can sort this list by country, etc.

I actually blocked a few large ranges yesterday, which would have blocked virtually all that traffic from Tuesday. And the same IPs were hitting me Monday as well. So you'd think things would improve, right?

But PERHAPS even if they're being blocked -- which wouldn't show up on any logs -- they're still keeping the router busy.

And here's the thing: right now some people CAN get into CathInfo. I could get in this morning (from my phone Internet -- so I'm coming from outside, as it were) but last night I couldn't. Looking at CathInfo itself, I see a list of members online all day long. So some people get lucky, while others get blocked by the traffic jam.

So we're right on the line -- which makes me think that a router 2X as fast would allow more legit members to "get lucky" and access the site.
If *no members* were getting in, one might suspect that a faster router would be futile, like trying to empty the ocean with a teaspoon.

Long story short, I don't care if gobs and gobs of excess server capacity is used (wasted) by bots. As long as all the members can get in, I don't care.
Logged

Offline Traditional Sermons

Re: CathInfo swarm of bot traffic
« Reply #6 on: February 19, 2026, 03:11:14 PM »
This is a growing concern with servers everywhere, with the rise of AI crawlers.
Some crawlers are more aggressive than others.

The best policy is to whitelist some of them, and then forbid any other "weird" behavior. A human being would not click through 30 pages in as many seconds, or look for admin login pages, so that kind of thing can be banned permanently.
Doing it that way means any human being can get through, which is what you want, while your site still gets known and listed, which is also good.

Ipset blocks the IPs before they even hit apache. Fail2ban and in particular custom filters is the security team that sends them there.

It would bring down also all the false positives on the forum pages.

It would be interesting to check out your CPU usage at the time of the the limited access. This is probably the cause of the timeouts and 500 errors. Because the server is getting choked up and drained.

Lets see how the new router changes things. It could indeed be a glitch in the firmware of the old one. Though its unlikely.
Logged

Offline Matthew

  • Mod
Re: CathInfo swarm of bot traffic
« Reply #7 on: February 19, 2026, 04:10:47 PM »
Quote from: Traditional Sermons on February 19, 2026, 03:11:14 PM
It would be interesting to check out your CPU usage at the time of the the limited access. This is probably the cause of the timeouts and 500 errors. Because the server is getting choked up and drained.

I keep a close eye on the server. The utilization scarcely goes over 4 or 5%, even when there are 12,000 "guests" on the forum, or when I can't access the forum from my phone (outside the house) Internet.

The server being choked is the 1st thing you check -- and I checked it.
Logged

Offline St Giles

  • Supporter
Re: CathInfo swarm of bot traffic
« Reply #8 on: February 19, 2026, 05:22:57 PM »
Some forums have relatively painful time limits set on certain actions: number of posts per xx amount of seconds, ect. It may be worth adding some limits such as 10 page requests per minute per IP, or a little time-out ban if someone uses too many resources too quickly, they have to wait 1-5 minutes before regaining access.
Logged

Offline Traditional Sermons

Re: CathInfo swarm of bot traffic
« Reply #9 on: February 19, 2026, 05:43:50 PM »
Quote from: Matthew on February 19, 2026, 04:10:47 PM
I keep a close eye on the server. The utilization scarcely goes over 4 or 5%, even when there are 12,000 "guests" on the forum, or when I can't access the forum from my phone (outside the house) Internet.

The server being choked is the 1st thing you check -- and I checked it.


Yes but the contrac tables could be near their limits on the server and that would not affect the CPU. Just run 
Code: [Select]
conntrack -CTo see current usage

And

cat /proc/sys/net/netfilter/nf_conntrack_max 


To see your limits.



Then there is the timeout values on the router which might need to be altered. If the attack is big enough, more RAM on a router won't make a difference when they have not been altered.
Logged
Pages: 1 [2] 3
« previous next »