WordPress xmlrpc.php bruteforce mitigation

The attacker 119.45.240.155 from my other post was also on the rampart blacklist.

Something changed at approximately 1900 yesterday:

2021-08-02 18:46:26,324 fail2ban.actions        [794198]: WARNING [evasive] 118.24.108.75 already banned
2021-08-02 18:48:25,674 fail2ban.actions        [794198]: NOTICE  [evasive] Unban 118.24.108.75
2021-08-02 18:48:33,890 fail2ban.filter         [794198]: INFO    [evasive] Found 118.24.108.75 - 2021-08-02 18:48:32
2021-08-02 18:48:33,893 fail2ban.actions        [794198]: NOTICE  [evasive] Ban 118.24.108.75
2021-08-02 18:48:47,895 fail2ban.filter         [794198]: INFO    [evasive] Found 118.24.108.75 - 2021-08-02 18:48:47
2021-08-02 18:48:47,927 fail2ban.actions        [794198]: NOTICE  [evasive] 118.24.108.75 already banned
2021-08-02 18:50:47,283 fail2ban.actions        [794198]: NOTICE  [evasive] Unban 118.24.108.75
2021-08-02 18:50:52,495 fail2ban.filter         [794198]: INFO    [evasive] Found 118.24.108.75 - 2021-08-02 18:50:51
2021-08-02 18:50:52,500 fail2ban.actions        [794198]: NOTICE  [evasive] Ban 118.24.108.75
2021-08-02 18:51:34,851 fail2ban.filter         [794198]: INFO    [evasive] Found 118.24.108.75 - 2021-08-02 18:51:34
2021-08-02 18:51:35,169 fail2ban.actions        [794198]: NOTICE  [evasive] 118.24.108.75 already banned
2021-08-02 18:52:12,979 fail2ban.filter         [794198]: INFO    [evasive] Found 118.24.108.75 - 2021-08-02 18:52:12
2021-08-02 18:52:13,219 fail2ban.actions        [794198]: WARNING [evasive] 118.24.108.75 already banned
2021-08-02 18:52:40,582 fail2ban.filter         [794198]: INFO    [evasive] Found 118.24.108.75 - 2021-08-02 18:52:39
2021-08-02 18:52:41,255 fail2ban.actions        [794198]: WARNING [evasive] 118.24.108.75 already banned
2021-08-02 18:53:43,979 fail2ban.filter         [794198]: INFO    [evasive] Found 118.24.108.75 - 2021-08-02 18:53:43
2021-08-02 18:53:44,538 fail2ban.actions        [794198]: WARNING [evasive] 118.24.108.75 already banned
2021-08-02 18:55:18,301 fail2ban.filter         [794198]: INFO    [evasive] Found 118.24.108.75 - 2021-08-02 18:55:17
2021-08-02 18:55:18,458 fail2ban.actions        [794198]: WARNING [evasive] 118.24.108.75 already banned
2021-08-02 18:55:35,927 fail2ban.filter         [794198]: INFO    [evasive] Found 118.24.108.75 - 2021-08-02 18:55:35
2021-08-02 18:55:36,483 fail2ban.actions        [794198]: WARNING [evasive] 118.24.108.75 already banned
2021-08-02 18:57:14,774 fail2ban.filter         [794198]: INFO    [evasive] Found 118.24.108.75 - 2021-08-02 18:57:14
2021-08-02 18:57:15,211 fail2ban.actions        [794198]: WARNING [evasive] 118.24.108.75 already banned
2021-08-02 18:57:56,145 fail2ban.filter         [794198]: INFO    [evasive] Found 118.24.108.75 - 2021-08-02 18:57:55
2021-08-02 18:57:56,467 fail2ban.actions        [794198]: WARNING [evasive] 118.24.108.75 already banned
2021-08-02 18:58:28,665 fail2ban.filter         [794198]: INFO    [evasive] Found 118.24.108.75 - 2021-08-02 18:58:28
2021-08-02 18:58:28,710 fail2ban.actions        [794198]: WARNING [evasive] 118.24.108.75 already banned
2021-08-02 18:58:44,187 fail2ban.filter         [794198]: INFO    [evasive] Found 118.24.108.75 - 2021-08-02 18:58:43
2021-08-02 18:58:44,730 fail2ban.actions        [794198]: WARNING [evasive] 118.24.108.75 already banned
2021-08-02 19:00:19,230 fail2ban.filter         [794198]: INFO    [evasive] Found 118.24.108.75 - 2021-08-02 19:00:18
2021-08-02 19:00:19,454 fail2ban.actions        [794198]: WARNING [evasive] 118.24.108.75 already banned
2021-08-02 19:02:18,204 fail2ban.actions        [794198]: NOTICE  [evasive] Unban 118.24.108.75
2021-08-02 19:06:14,462 fail2ban.filter         [794198]: INFO    [evasive] Found 8.209.68.40 - 2021-08-02 19:06:13
2021-08-02 19:06:14,518 fail2ban.actions        [794198]: NOTICE  [evasive] Ban 8.209.68.40
2021-08-02 19:08:13,286 fail2ban.actions        [794198]: NOTICE  [evasive] Unban 8.209.68.40
2021-08-02 19:59:53,539 fail2ban.filter         [794198]: INFO    [evasive] Found 167.172.24.96 - 2021-08-02 19:59:53
2021-08-02 19:59:53,599 fail2ban.actions        [794198]: NOTICE  [evasive] Ban 167.172.24.96
2021-08-02 20:01:53,769 fail2ban.actions        [794198]: NOTICE  [evasive] Unban 167.172.24.96
2021-08-02 23:39:39,784 fail2ban.filter         [794198]: INFO    [evasive] Found 62.210.140.247 - 2021-08-02 23:39:38
2021-08-02 23:39:40,569 fail2ban.actions        [794198]: NOTICE  [evasive] Ban 62.210.140.247
2021-08-02 23:41:38,735 fail2ban.actions        [794198]: NOTICE  [evasive] Unban 62.210.140.247
2021-08-03 00:09:10,270 fail2ban.filter         [794198]: INFO    [evasive] Found 35.223.6.199 - 2021-08-03 00:09:10
2021-08-03 00:09:10,856 fail2ban.actions        [794198]: NOTICE  [evasive] Ban 35.223.6.199
2021-08-03 00:11:10,225 fail2ban.actions        [794198]: NOTICE  [evasive] Unban 35.223.6.199

There were previously lots of warnings from evasive about IPs “already banned” and then after 1900 the warnings ceased, and neither recidive or evasive reported any more activity from those IP addresses shown at the end of the above log. Evasive actually hasn’t reported a single action or filter today. So maybe it just took time to kick in? Happy to see that the attacks have stopped.

I don’t know if this is helpful, but I set the variables for recidive_maxretry, etc manually since I found that they were not set when I attempted

That’s normal. Those defaults are inherited from the respective role. Setting via cp.bootstrapper overrides those respective defaults.

Did those attacks cease after 17:12:34 when the attacker was upgraded to recidive? Apache uses buffered logs to reduce synchronous writes. Depending upon volume, these may take a minute or two to write to disk. What I’m seeing is consistent with that I’d expect,

2021-08-03 04:01:06,596 fail2ban.filter         [22978]: INFO    [evasive] Found 209.126.13.151 - 2021-08-03 04:01:05
2021-08-03 04:24:24,276 fail2ban.filter         [22978]: INFO    [evasive] Found 138.197.131.66 - 2021-08-03 04:24:23
2021-08-03 04:34:47,570 fail2ban.filter         [22978]: INFO    [evasive] Found 162.241.131.11 - 2021-08-03 04:34:46
2021-08-03 04:42:01,500 fail2ban.filter         [22978]: INFO    [evasive] Found 66.33.212.126 - 2021-08-03 04:42:00
2021-08-03 04:52:03,339 fail2ban.filter         [22978]: INFO    [evasive] Found 67.205.11.196 - 2021-08-03 04:52:02
2021-08-03 05:09:57,522 fail2ban.filter         [22978]: INFO    [evasive] Found 34.145.79.165 - 2021-08-03 05:09:56
2021-08-03 05:17:55,488 fail2ban.filter         [22978]: INFO    [evasive] Found 173.236.184.100 - 2021-08-03 05:17:54
2021-08-03 05:28:36,426 fail2ban.filter         [22978]: INFO    [evasive] Found 67.205.42.180 - 2021-08-03 05:28:34
2021-08-03 05:30:06,355 fail2ban.filter         [22978]: INFO    [evasive] Found 147.182.164.34 - 2021-08-03 05:30:04
2021-08-03 06:02:30,480 fail2ban.filter         [22978]: INFO    [evasive] Found 143.198.69.89 - 2021-08-03 06:02:29
2021-08-03 06:40:59,652 fail2ban.filter         [22978]: INFO    [evasive] Found 208.113.217.164 - 2021-08-03 06:40:58
2021-08-03 07:12:08,022 fail2ban.filter         [22978]: INFO    [evasive] Found 195.154.200.175 - 2021-08-03 07:12:07
2021-08-03 07:21:02,639 fail2ban.filter         [22978]: INFO    [evasive] Found 128.199.56.61 - 2021-08-03 07:21:01
2021-08-03 07:21:24,874 fail2ban.filter         [22978]: INFO    [evasive] Found 159.75.118.109 - 2021-08-03 07:21:24
2021-08-03 07:34:36,755 fail2ban.filter         [22978]: INFO    [evasive] Found 167.172.37.241 - 2021-08-03 07:34:36
2021-08-03 07:37:02,309 fail2ban.filter         [22978]: INFO    [evasive] Found 147.182.164.34 - 2021-08-03 07:37:00
2021-08-03 07:50:53,710 fail2ban.filter         [22978]: INFO    [evasive] Found 23.253.236.94 - 2021-08-03 07:50:53
2021-08-03 07:52:57,930 fail2ban.filter         [22978]: INFO    [evasive] Found 23.253.236.94 - 2021-08-03 07:52:57
2021-08-03 07:56:51,498 fail2ban.filter         [22978]: INFO    [evasive] Found 95.181.152.71 - 2021-08-03 07:56:50
2021-08-03 08:12:35,187 fail2ban.filter         [22978]: INFO    [evasive] Found 128.199.55.11 - 2021-08-03 08:12:33
2021-08-03 08:48:56,245 fail2ban.filter         [22978]: INFO    [evasive] Found 35.222.152.162 - 2021-08-03 08:48:55
2021-08-03 08:54:21,158 fail2ban.filter         [22978]: INFO    [evasive] Found 157.245.68.43 - 2021-08-03 08:54:19
2021-08-03 10:16:25,286 fail2ban.filter         [22978]: INFO    [evasive] Found 89.187.164.166 - 2021-08-03 10:16:24
2021-08-03 10:26:30,240 fail2ban.filter         [22978]: INFO    [evasive] Found 194.26.29.21 - 2021-08-03 10:26:29
2021-08-03 10:30:22,670 fail2ban.filter         [22978]: INFO    [evasive] Found 137.117.83.116 - 2021-08-03 10:30:22

It’s processing ~2.5 req/sec according to systemctl status httpd.

cpcmd scope:set apache.buffered-logs false

You’ll take an IO hit but on a smaller server the overhead is negligible :+1:

fail2ban and evasive seem to be functioning well now. I also added a custom configuration for xmlrpc.php as shown by the docs here: Brute-force protection | ApisCP Docs

Please let me know if the large DOSPageInterval makes sense or not. I wanted to be stricter about attacks that only make one request per minute. I think many plugins which currently use xmlrpc.php are probably switching to use the WordPress REST API anyway.

# Block xmlrpc.php brute-force attempts
<Files "xmlrpc.php">
    <If "%{REQUEST_METHOD} != 'POST'">
        DOSEnabled off
    </If>
    DOSPageCount 3
    DOSPageInterval 300
</Files>

Look at its function, as a DoS prevention measure. That means you’re focused on stopping high-flow attacks that can swamp system resources or facilitate vulnerability discovery. If you’re seeing 1 attempt per 60 seconds, then it’s immaterial for a few reasons:

  1. The cost of tracking a connection over a 5 minute window is impractical. There’s a finite number of entries tracked. You can increase this value; however, the hash table lives and dies with the HTTP worker.
  2. Workers are limited to 1024 requests before spinning up a new worker. For that request to be effectively tracked
  3. Server runs multiple workers. Keepalive reuses the same connection slot (and worker) iff a subsequent request arrives within 3 seconds. If it’s outside of that window, then it may arrive on any other worker depending upon availability. That means a new hash entry for that worker and if the count is 2 on worker 1 and 0 on worker 2, then you could reasonably hit (limit - 1) x workers + 1 requests until pigeonhole principle prevails, where limit is DOSPageCount and workers is the number of HTTP workers.
  4. There are legitimate use cases for pingbacks. If you want to disable xmlrpc entirely, then consider an .htaccess rule,
    <Files xmlrpc.php>
     order deny,allow
     deny from all
     allow from 127.0.0.1
    </Files>
    
  5. Let’s say it’s a password cracking situation and your password is hunter3 to avoid being the top 1000 passwords. That means an attacker has to permute a space of about 36^7 combinations. At a rate of 1 request a minute, it’d take 36^7/(60 x 24) ~ 54419558 days to work it out. 1 request a minute is nothing to be alarmed over.

Avoid quixotic conquests to achieve perfectly clean logfiles. It’s never practical unless you completely airgap your server by putting it on a local network with zero outside access. Use the features that are available as intended and always keep your plugins updated.

1 Like