About a month ago one of our servers which hosts some WordPress sites suffered from very high load. This was really unsual because our server often does not work much at nights. Being alerted by New Relic that the Appdex score falled down to 0.5 at about 10PM, I logged into the server to see what was happening, and did not forget check New Relic dashboard. It reported that some of the sites were using so much CPU and memory. What was going on? Something smelled fishy. First I checked the access logs:

tailf log/access.log

And what I got were a lot lines like these:

...
87.160.26.38 - - [07/Nov/2013:22:32:23 +0700] "POST /wp-login.php HTTP/1.0" 200 4461 "awpsite.com/wp-login.php" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:22.0) Gecko/20100815 Firefox/22.0"
88.103.119.203 - - [07/Nov/2013:22:32:32 +0700] "POST /wp-login.php HTTP/1.0" 200 4461 "awpsite.com/wp-login.php" "Opera/9.80 (Macintosh; Intel Mac OS X 10.5.1; U; ru) Presto/2.3.42 Version/11.10"
190.41.24.189 - - [07/Nov/2013:22:32:40 +0700] "POST /wp-login.php HTTP/1.0" 200 4461 "awpsite.com/wp-login.php" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:21.0) Gecko/20090815 Firefox/21.0"
180.234.69.192 - - [07/Nov/2013:22:32:43 +0700] "POST /wp-login.php HTTP/1.0" 200 4461 "awpsite.com/wp-login.php" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_1_2; en-US) AppleWebKit/593.16 (KHTML, like Gecko) Chrome/10.0.599.250 Safari/401.16"
109.63.89.180 - - [07/Nov/2013:22:32:48 +0700] "POST /wp-login.php HTTP/1.0" 200 4464 "awpsite.com/wp-login.php" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/6.0)"
109.63.89.180 - - [07/Nov/2013:22:32:49 +0700] "POST /wp-login.php HTTP/1.0" 200 4464 "awpsite.com.com/wp-login.php" "Opera/9.80 (Windows NT 6.1; U; ru) Presto/2.1.64 Version/10.10"
181.64.181.65 - - [07/Nov/2013:22:33:40 +0700] "POST /wp-login.php HTTP/1.0" 200 4463 "awpsite.com/wp-login.php" "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/6.0)"
181.64.181.65 - - [07/Nov/2013:22:33:41 +0700] "POST /wp-login.php HTTP/1.0" 200 4462 "awpsite.com/wp-login.php" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"
181.64.181.65 - - [07/Nov/2013:22:33:42 +0700] "POST /wp-login.php HTTP/1.0" 200 4461 "awpsite.com/wp-login.php" "Mozilla/5.0 (Windows NT 6.1; rv:22.0) Gecko/20120101 Firefox/22.0"
...

There were so many login attempts from a lot of IP addresses. It looked like that our WordPress sites were the targets of a brute force attack. I was wondering how these nuisances could be stopped, especially when they came from so many sources. In fact, in case an attack like this succeeds, it is still difficult for the attacker to get access to our WordPress admin page as we still have some more extra protection layers. At this point I needed to have a closer look at how the attacker was playing with my sites so I used tcpdump to capture ongoing packets:

tcpdump -i eth0 -w dump.cap

I waited about one minute, terminated tcpdump and downloaded the file to my computer to analyse the packets with Wireshark. The file was not too large but contained what I needed. There were so many packets, but what I was interested in is what was being posted to wp-login.php page, and they all looked like this:

Screenshot from 2013-12-08 15:45:16

All these POSTs had different user agents but same referer. They were trying to guess the password of the user “admin”. If this is the username of the administrator and he uses a simple password, his account will soon be cracked. I wanted to know that if a legitimate login had the same characteristics, so that I could filter out these illegal logins. I used tcpdump again and tried to login to the site:

tcpdump -w dump.cap -i eth0 dst my_server_ip and port 80

Actually, my login looked much more legitimate:

Screenshot from 2013-12-08 15:39:47

An illegal login does not come with some headers: Accept, Accept-Language, Accept-Encoding and Cookie. Also, it lacks a wp-submit field in the POST values. At this point, I decided to filter out the logins which did not have the field wp-submit. And I needed to apply this to all the WordPress sites, not only a specific one so I chose to use Snort. Snort is an open source Instrusion detection system, which can be used to block and alert these type of attacks effectively. I spent nearly an hour to get Snort up and running on my server. Writing a Snort rule was not very complicated. Here was the rule I made:

alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"Wordpress Brute Force Login"; flow:to_server,established;content:"POST"; nocase; http_method; uricontent:"/wp-login.php"; nocase; content:!"wp-submit"; nocase;  classtype:web-application-attack; react:block; sid:20000100; rev:1;)

The rule tells Snort to check incoming packet which has established a connection with the web server before, and has “POST” in its content, also has an uri as “/wp-login.php”, and doesn’t have “wp-submit” in its content. Snort will block these packets if they match this rule.

I added this rule to

/etc/snort/rules/web-attacks.rules

Then checked if the syntax is ok:

/usr/local/bin/snort -u snort -g snort -c /etc/snort/snort.conf -i eth0

Then I restarted snort:

/etc/init.d/snortd restart

Yeah, it was time to check if Snort worked as expected. But unfortunately, at this time, the attack had been stopped, there was no more log entries of the illegal logins in access logs. I had to check it out by myself by submiting fake information, without a wp-submit field, of course:

Screenshot from 2013-12-08 16:05:35

Voila! The login was forbidden with a 403 error. Snort works as expected. Here is what was reported in Snort’s log:

Screenshot from 2013-12-08 16:10:03

The attack was continued the after days, and keep going on but Snort could handle them. That’s it.