Author Topic: CathInfo swarm of bot traffic (Read 200 times)

Matthew · « **on:** **Yesterday** at 06:53:39 PM »

I am aware of the problem where some visitors can't access the site. The best I can tell, it has to do with a swarm of bot traffic hitting the site constantly.

I am currently investigating and continuing to pursue what options I have to mitigate this problem.

I was made aware of the problem yesterday, and that's when I began trying things. Today I went 8 minutes away and tried to visit CathInfo from my mother-in-law's computer. Totally timed out.
At that point, I moved it up to #1 priority.

I will make it my #1 priority to fix, perhaps after important family duties and personal care.

I do know that the server is not the problem; most of the time it hovers around "1.0" when the max is "24.0".

And I can visit the site at any time from home; the software reports being able to build the page in a fraction of a second. So it's something on the network side of things. I still need to figure out the exact bottleneck, where the traffic jam is happening.

Matthew · « **Reply #1 on:** **Yesterday** at 08:38:06 PM »

Having done some research, I know a bit more now.

The CathInfo server is actually 3 1/2 years old (which is fine; the server isn't the problem) but the router is just as old. I wanted to know if they make better ones (more CPU, more RAM, to handle more connections per second) in the last 3.5 years.

Turns out, they do.

https://shop.asus.com/us/90ig08g0-ma1a0v-rog-rapture-gt-be98-pro.html

* I know the server is fine. Pages are created near-instantly on my end. The server isn't running that hard.
* I know my Internet connection is fine (300 Mbps up/down) because I can do whatever I want on the Net with no slowdown. So my Internet pipe isn't clogged.
So my best guess is that the router is slowing things down during the forwarding of traffic from the "entrance to my house" to the CathInfo server.

I could be wrong, but that is my best guess.

From what I know about computers (and my knowledge is quite broad, and sometimes deep as well) it is possible that port forwarding work is done by just one of the "cores" of the CPU. So the faster the cores are, the faster that traffic can be processed.
Also, it's possible that ALL the memory of the unit can't be dedicated to "active connections on the router", although in an ideal world it would be.
So if you double the RAM, chances are, the manufacturer's engineers would (hopefully) increase whatever "setting" limits how much RAM is allocated to port forwarding.

Just an educated theory.

This router has a 2.6 GHz quad-core CPU, and 2 GB of RAM.
The existing router has 1.8 GHz quad-core CPU and 1 GB of RAM.

TL;DR -- this new router is $630. So if someone wants to help out with this, I would be happy to install it.

Traditional Sermons · « **Reply #2 on:** **Today** at 05:14:03 AM »

I asked Claude about this and to be honest I agree with the assessment. Your router should not be causing the issue:

"The router almost certainly isn't your bottleneck here. Even an older consumer router can handle NAT/port forwarding for thousands of concurrent connections without breaking a sweat. What's more likely happening is that bot traffic is exhausting your router's connection tracking table (conntrack) — and buying a faster router just
lets you process junk traffic faster. It doesn't stop the junk from arriving.

The fix is to drop bad traffic before it becomes a problem. Here's what I'd suggest, in order:

1. ipsets + fail2ban on the server

ipsets let you maintain large blocklists in the Linux kernel. Lookups are near-instant even with hundreds of thousands of entries. fail2ban watches your web server logs and automatically adds abusive IPs to the blocklist. Packets get dropped at the kernel level before your web server ever sees them.

You can also bulk-load known bad IP ranges (datacenters, cloud providers, known bot networks) into ipsets. Most legitimate visitors aren't coming from AWS or DigitalOcean IP ranges.

2. Rate limiting at the firewall level

iptables/nftables can rate-limit new connections per source IP. Something like 20-30 new connections per minute per IP is reasonable for a forum. Real users won't hit that limit; bots will.

3. Check your conntrack table

Run cat /proc/sys/net/netfilter/nf_conntrack_max on the server. If bots are filling this up, you can increase it. Also run conntrack -C to see how many entries are currently active. This will tell you if that's actually the bottleneck.

4. Then reassess the router

After doing the above, you may find the problem is completely solved. If not, and the router is still choking, a MikroTik or a small pfSense/OPNsense box would be a much better investment than a $630 gaming router. Those gaming routers are optimized for WiFi and game traffic QoS — features that do nothing for serving a website. A
$50-100 MikroTik would outperform it for this specific use case.

The short version: block the bad traffic first, then see if you still have a problem. No point in buying a bigger pipe if you can just shut off the firehose."

Traditional Sermons · « **Reply #3 on:** **Today** at 05:19:55 AM »

If you need help with this, let me know.

Matthew · « **Reply #4 on:** **Today** at 07:50:00 AM »

Update:
I ordered the new router. I need to do some preparation work until it arrives -- there are a lot of settings that need to be carried over.

Considering it involves my very connection to the Internet, I should wait until Monday (a business day, when my ISP is open for business) to do the switchover, in case there's any "trouble".

But for something like this (new server, or new router) the prep work is a huge chunk of the work of upgrading.

Matthew · « **Reply #5 on:** **Today** at 07:55:51 AM »

In response to TraditionalSermons --

I didn't mention all the work I did with the server logs. I have the ability to look at an aggregate every day, showing the top 366 (yes, that exact number) IP addresses that hit the site. I can sort this list by country, etc.

I actually blocked a few large ranges yesterday, which would have blocked virtually all that traffic from Tuesday. And the same IPs were hitting me Monday as well. So you'd think things would improve, right?

But PERHAPS even if they're being blocked -- which wouldn't show up on any logs -- they're still keeping the router busy.

And here's the thing: right now some people CAN get into CathInfo. I could get in this morning (from my phone Internet -- so I'm coming from outside, as it were) but last night I couldn't. Looking at CathInfo itself, I see a list of members online all day long. So some people get lucky, while others get blocked by the traffic jam.

So we're right on the line -- which makes me think that a router 2X as fast would allow more legit members to "get lucky" and access the site.
If *no members* were getting in, one might suspect that a faster router would be futile, like trying to empty the ocean with a teaspoon.

Long story short, I don't care if gobs and gobs of excess server capacity is used (wasted) by bots. As long as all the members can get in, I don't care.

Author Topic: CathInfo swarm of bot traffic (Read 200 times)

Matthew

CathInfo swarm of bot traffic

Matthew

Re: CathInfo swarm of bot traffic

Traditional Sermons

Re: CathInfo swarm of bot traffic

Traditional Sermons

Re: CathInfo swarm of bot traffic

Matthew

Re: CathInfo swarm of bot traffic

Matthew

Re: CathInfo swarm of bot traffic