Best practice blocking malicious URL encoded attacks

Support
1

Some of my clients are victim of an attack from multiple IPs all over the world - Never exceeds 1-4 request per second. Same IP usually visits again after few hours.

Below is a snippet of accesslog, now these request are changing dynamically with another lenght or some few other parameters, but i would be able to create a pattern-list and block them boringly that way. I tried actually using fail2ban by creating a filter and jail.conf, but i am not sure this is the correct approach or other better alternate solution exist (CF is not an option here)

How can one fight against these type of attacks? Is fail2ban the right way? If so, is it as simple as creating the jail and filter.conf with correct regex and place them in jail.d/ and filter.d/ folders?

"GET /4-gift?order=product.name.asc&pris=20-500%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.name.asc?order=product.name.asc&pris=20-500%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.name.asc HTTP/1.1" 200 43255 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.2151.63 Safari/537.36"
"HEAD /10-presentgifts?order=product.random.desc&page=9&pris=150-300%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.random.desc?order=product.random.desc&page=9&pris=150-300%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.random.desc HTTP/1.1" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.2598.44 Safari/537.36"
"GET /10-presentgifts?order=product.price.asc&page=2&pris=150-300%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc?order=product.price.asc&page=2&pris=150-300%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc HTTP/1.1" 200 42608 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.2916.43 Safari/537.36"
"GET /4-gift?order=product.name.asc&page=60&pris=20-500%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc HTTP/1.1" 301 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3583.125 Safari/537.36"
"GET /4-gift?order=product.quantity.desc&pris=20-300%3Fpage%3D32%3Fpage%3D32%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.quantity.desc HTTP/1.1" 301 - "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.2597.181 Safari/537.36"
"GET /4-gift?order=product.price.desc&pris=20-300%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc HTTP/1.1" 301 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.2915.44 Safari/537.36"
"GET /4-gift?order=product.price.desc&pris=20-300%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc?order=product.price.desc&pris=20-300%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.random.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.position.asc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.name.desc%3Forder%3Dproduct.date_add.desc%3Forder%3Dproduct.quantity.desc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.price.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.price.desc%3Forder%3Dproduct.name.asc%3Forder%3Dproduct.name.asc HTTP/1.1" 200 43334 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.2915.44 Safari/537.36"
2

You may consider LimitRequestLine available in server config and virtual host contexts.

For something delegated to the site’s authority in .htaccess,

RewriteEngine on
RewriteCond %{QUERY_STRING} ^.{256,}
RewriteRule ^ - [R=414,L]

Respond with 414 URI Too Long if query string exceeds 255 characters.

Another consideration is whether 1-4 requests every few hours is enough CPU draw to warrant mitigation.

3

Not bad advice actually. The lenght might save me for this specific attack. I really dont want to use .htaccess method - too “much” performance impact, but server conf i will try out.

1-4 req is PER ip - They are attacking with thousands IP… So i have constant hits. Load is increased x10

At the end i would rather want it to block on kernel/network level by utilizing fail2ban etc.

conf location: /etc/fail2ban/jail.d

custom-bot-filter]
enabled  = true
filter   = custom-bot-filter
logpath  = /home/virtual/site2/shadow/var/log/httpd/access_log
maxretry = 1
bantime  = 2592000
banaction=firewallcmd-ipset-reset
backend=pyinotify
banaction_allports=firewallcmd-ipset[actiontype=""]
protocol=tcp

Filer location: /etc/fail2ban/filter.d

[Definition]
failregex = ^(?P<host><HOST>) .* "GET .*page%%3D\d+%%3F(?:page%%3D\d+%%3F)+page.*"
ignoreregex =

EDIT: Bingo, I can see it works now with the regex, and i also see blocks - so far so good.
fail2ban-client status custom-bot-filter =

Status for the jail: custom-bot-filter
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     48
|  `- File list:        /home/virtual/site2/shadow/var/log/httpd/access_log
`- Actions
   |- Currently banned: 1
   |- Total banned:     1
   `- Banned IP list:   4.227.36.92

Only issue is that the ban is only effective after a firewall reload - which doesnt happen automatically. Which setting do i miss in my conf ?
EDIT: No, it doesnt block even after firewall-reload. I still see the ip hitting my servers.
EDIT 2: Now fixed with new jail conf - All good! Blocking works!

Question 1:
How can I include all accesslogs in the logpath, so this bot-filter works for all clients/accounts on server ?
Do i need to include something like /home/virtual/*/shadow/var/log/httpd/access_log ?

Question 2:
Is the above solution correct approach and sufficient ? Without conflicting with apiscp structure etc ?

4

Here’s a resource that I found that may be of use: how to specify multiple log files pattern in fail2ban jail? - Stack Overflow

It may need to be refined to /home/virtual/*/shadow/var/log/httpd/access_log{,.1} to include rotated logs - I’d play around with it to see what works best.

No conflict. You can verify this to be the case by performing a platform scrub for the relevant roles:

upcp -sb network/setup-firewall fail2ban/configure-jails

A full scrub (upcp -sb) would be more comprehensive at the expense of runtime.