• Author
• Recent Posts

Guest Poster

This post was contributed by a guest poster. More details below.

Latest posts by Guest Poster (see all)

• A Judge’s Perspective: How Shared Calendars Can Reduce Legal Custody Issues - March 17, 2016
• 13 Essential Phone and iPad Apps to Help You Stay Sane During Your Move - November 6, 2012
• 5 Tips For Surviving With a WiFi-only Tablet - November 2, 2012

Editor’s note: Today, 40Tech is pleased to present you with a guest post from Lazy Man of Lazy Man and Money.This article is intended to demonstrate one man’s thoughts on what was happening during a denial of service attack, and how he dealt with it.

The second week in February was a very bad week for me. On February 6th, I had received a legal threat from LifeVantage regarding my ProtandimScams.com site. I was still crushed by my beloved Patriots losing the Super Bowl. (Hey, I put up with their 1-15 seasons and Lisa Olsen scandals, so I’m milking the Tom Brady era for all it is worth). On the 8th, my websites stopped working. I went to my Putty window running a Unix top command to see what was the matter. The load average had spiked from its normal level of around 0.50 to 120. If you aren’t familiar with Unix, Top, or Putty, this means that either something on your site isn’t working right or Yahoo decided to feature you on its home page. Here’s what happened next.

I had to shutdown my webserver (Apache) and database (mysql). I took a look at Amazon’s control panel and saw this:

The usual traffic load for my website is about 2 million bytes out. Suddenly it is at 200 million. That means I just received 100 times more traffic than the norm. I looked to see if I had made the homepage of Yahoo, but there were no unusual referrals from Sitemeter, one of my website stat keeping programs. What I did see was the following:

64.150.238.202 – – [03/Feb/2012:22:57:49 +0000] “POST /contact/ HTTP/1.1” 200 43826 187.127.234.99 – – [03/Feb/2012:22:57:48 +0000] “GET / HTTP/1.0” 200 64441 217.12.59.180 – – [03/Feb/2012:22:57:48 +0000] “GET / HTTP/1.0” 200 64441 190.51.72.55 – – [03/Feb/2012:22:57:48 +0000] “GET /contact/ HTTP/1.1” 200 46029 201.8.193.241 – – [03/Feb/2012:22:57:47 +0000] “GET / HTTP/1.0” 200 64441 78.100.197.115 – – [03/Feb/2012:22:57:48 +0000] “POST /contact/ HTTP/1.1” 200 43826 190.229.178.135 – – [03/Feb/2012:22:57:49 +0000] “GET /contact/ HTTP/1.0” 200 44306

In a single second, and every second I had around 75 such requests. I was viewing the log with a “tail -f” unix command; it almost looked like the Matrix it was scrolling so fast. I instantly suspected a distributed denial of service (DDoS) attack. If you aren’t familiar with a DDoS, I wrote a layman’s description of one here. I asked a few friends and they looked at the IP addresses and confirmed that they were mostly coming from small countries in the Middle East or ex-Soviet Union. This is not the main audience of Lazy Man and Money.

It didn’t surprse me that someone would want to attack me with a DDoS. I’ve been outspoken on certain topics, such as asserting that MonaVie, Protandim, and other multi-level marketing schemes are scams. [Editor’s note: we’re not familiar with any of these products, and thus can’t represent whether they are in fact scams or not – you’re urged to look into this yourself and draw your own conclusions. We also don’t know whether they or some other entity had anything to do with the attack.].

My first step was to get my webite up and going. Amazon Web Services had recently allowed consumers to spin up a legit super computer on demand. So that’s just what I did in launching their “Cluster Compute Eight Extra Large” which has two Intel Xeon processors (16 cores) and 60GB of RAM. What I was running previously had about 1/20th the processing power and only 1.7GB of RAM. The brute force attempt worked… mostly. I could keep my website up, but it was slow and it timed out quite a bit.

Brute force wasn’t the solution. I needed to do something smarter. I decided to move my contact page to a different URL (as they were targeting it) and send anyone looking for it, a 410 Gone from my .htaccess file. On a recommendation from a friend, I also sent all HTTP 1.0 traffic a 410 Gone as well, since all major browsers have used HTTP 1.1 for awhile now. This would reduce the amount of work my site had to do serving the traffic.

It was at this point that I had my first big surprise. I had put the code on my Lazy Man and Money website and it did nothing. It just didn’t make sense. Lazy Man and Money is the site with all the search rankings that these companies hate. It is also where the vast majority of my income comes from. They had to be attacking that site, right? I was wrong. Previously, I had all the logging of my websites funnel to one log file. It took me an hour to think to put the logging of each website’s traffic in different files. Then it became clear. The attackers were going after my JuiceScam.com site. Once I put the blocks in there things got a little better, but it was still only solving about 10% of the problem.

It’s probably a little late to introduce this fact, but I’m not an Linux administrator. I took on the role because Amazon gives you a great deal of computing power for the money and Dreamhost’s virtual private servers were no longer economical or reliable for me. From being a software engineer at a couple of start-ups, I wore the Linux admin hat part-time on occasion and as a change of pace, it was a little fun. (Yes, I’m a big geek/dork.) If I were a real Linux admin, I probably could have solved this myself in a few minutes.

I looked forward to the challenge. It was something that was very different from my typical blogging day. I went through a number of failed attempts. I installed the Apache module mod_evassive, only to find that it didn’t work for my particular attack. There were tens of thousands of IP addresses rotating their attack over the course of a few minutes, so I couldn’t just block a few repeating every minute. I next tried to work unix iptables, but that was outside my skill set. I installed the Bad Behavior plugin for WordPress and use Project Honeypot’s list of bad IP addressed, hoping that would work, it didn’t help much.

Finally, I gave up and said, “It’s time to get a professional.” I asked a bunch of friends about their favorite hosting companies. Many people swore by LiquidWeb. I told them my situation and they politely told me to take my business elsewhere. I then looked to companies that help with those who have fallen victim to DDoS attacks. Some of them included Black Lotus, Prolexic, and Verisign. The latter two got back to me at prices that would be equivalent to around $30,000-50,000 a year. That’s a lot of money for someone just trying to do a some good and call light to an issue. Black Lotus was going to be expensive, too, but at least it seemed like a reasonable option, as long as I didn’t need a dedicated server.

One friend mentioned WP Engine, a company I had never heard of.

The Solution to the Problem

I looked at WP Engine’s website and it was almost too good be true. They promised DDoS protection, but they went beyond that. I saw that they banned many popular WordPress plugins. Why? Because they slow down websites too much. I realized that this hosting company was a little different. They only do WordPress and they put every effort into optimizing it. Best yet, is their slogan, “Quit worrying and let us run WordPress for you.” This is what I need, but I had my doubts.

I emailed their sales group and explained my situation. They responded with

“So we can definitely handle DDOS without an issue — in fact, our firewall can block 1.5M hits/sec while still allowing normal traffic through. And our retained security firm is constantly monitoring and updating the bad-actor IP block list. We’ve also got a stellar anti-malware setup.”

That was certainly encouraging. Being able to block 1.5 million hits a second would surely make the 75 I was dealing with look like an ant attack an elephant. At $100 a month for the professional plan (allow me 10 sites totaling 100,000 hits a month), it was a reasonable price too. All I needed was for it to live up to its hype.

I signed up for WP Engine (they have a 15-day money back guarantee) figuring that it was worth a shot. At 7AM on my vacation in Maui, I pressed the button and transfered my traffic to the WP Engine’s servers. I went to refresh my site and WP Engine returned a screen that their websever was down. It was only a couple of minutes before I got an email from WP Engine:

“So the DDoS is actually *really* bad. Even with our firewalling and security rules, it’s chewing up the resources on a whole cluster by itself. And that’s disturbing, because we had a customer earlier in the week that was handling 27,000+ simultaneous connections over the course of several hours without any issues at all.”

They went to work on figuring out what going wrong on their end, but suggested that I sign up with Cloudflare, a company that claims to optimize your site and protect it from DDoS. The idea was to stack multiple levels firewall defense. You set the name servers to Cloudflare and they, in theory, filter out the bad stuff before it even gets to the hosting. Ironically, just a few hours before someone had read about my DDoS and left a comment suggesting that I use CloudFlare as well. Since CloudFlare is only $20 a month for a pro account, I splurged on that. I figured I could also go down to the free account later, but let’s get every horse pulling to make it work. Cloudflare is supposed to learn to filter out attackers automatically. The only problem was that Cloudflare wasn’t learning… it was stopping about 1% of the attacks. However, Cloudflare does allow you to manually “block” countries. I put block in quotes, because visitors of blocked countries get a captcha to make sure they are human. They can still get to the website. I started blocking the 200+ countries that I typically don’t get any typical traffic from (Sorry Borat, Kazakhstan was one of the first to go).

While this was going on the people at WP Engine were working around the clock. They found that I had a WordPress plugin, Who Sees Ads, that was breaking their caching technology. When they disabled that plugin, my website was back functioning 100%. There was still an attack going on, but visitors to my website would never have known it. While I continued to block sites, the people at WP Engine worked on blocking the worst offenders that were sneaking through. In a few hours we had eliminated what turned out to be 200 hits a second at its peak to just about nothing:

Interestingly, my Amazon server was still getting attacked. I thought it was because the DNS just didn’t fully propagate, but after a day, I realized something just wasn’t right. Playing a hunch, I put one of my other lightly trafficked websites first in my virtual host line-up and it suddenly got traffic. This told me that part of the attack was aimed directly at the IP of my Amazon server. This is where having Amazon Web Services really paid off. I simply grabbed a new elastic IP and attached my server to that, taking all of 30 seconds. I then updated my sites in Cloudflare (I moved everything there now) to point to this new IP address. I discarded the one that was being attacked.

I can’t say enough about WP Engine. They worked round the clock to get the site up and running and I have proof that (as long as I don’t install plugins that break their caching system) it can survive any rush of traffic that I’ll ever see. Just to show the difference it made, here’s what my Google Webmaster Tools shows for site speed (keep in mind that this was with the switch to Amazon’s most powerful server):

Have you ever been subject to a denial of service attack? How have you handled it?

This article is written by Lazy Man who writes for Lazy Man and Money, a successful personal finance site. In addition to that personal finance site, he runs a few consumer-oriented sites including MonaVie Scam and Protandim Scams. He has previously written here about how his website crashed and burned with Amazon Web Services Outage on April 19, 2011.