At about 9:00 AM yesterday morning, the web design firm I am working for started experiencing some issues with the servers that host most of their websites. At first it was pretty standard because occasionally one of the servers will act up a bit, but will resume as normal. It started out with just one server going down, and then the other ones followed and stopped operating correctly as well. It was pretty clear at that point that something had gone wrong. One of the owners of the company decided to take a drive to the facility that houses the company’s web servers and most other networking equipment, called nFrame in Carmel, IN. I have gotten the chance to be at nFrame a few times now when there was a need for changes in the equipment or software, and now for this DDoS. The nFrame facility is a networking enthusiast’s dream, and has several different rooms lined with racks of networking equipment, large cooling units, removable tiled floor to store wires under, and very high security procedures including biometrics.
After investigation, it was determined that one particular server was being bombarded by ICMP requests by many different IP’s. Furthermore, it was narrowed down to a “SYN” flood attack. “SYN” is a part of the acknowledgment process in packets that use TCP/IP networking. Since the flood of requests was originating from many different IP addresses, it can be considered a “distributed” denial of service attack. This is likely the works of a botnet with the botmaster being located in another country. Since it was the work of a botnet, it would be nearly impossible to find out the true orgin of the DDoS. That is the crazy thing about botnets, is that most of these ICMP requests came from computers that don’t even know are doing they sending requests, but have been infected and are running underlying malicious software.
The attacks were overloading the router that all the servers used to route the incoming traffic, which in turn made all of them crash. Putting a hardware firewall in there would have been the best fix, but unfortunately no firewalls were readily available. It was a very aggressive attack, at the peak of traffic sending in about 2 million requests every 40 seconds. There were several attempted fixes that were put into place such as blocking ranges of IP’s in the router that were originated in other countries. Unfortunately the IP range of the requests were being changed by the attack every 20 minutes and it didn’t really help as much as hoped. One of the owners even made a PHP script to send out an email containing the unique range of IP’s every 20 minutes to be input into the router IP filter. The only problem here is that someone had to manually enter in the IP ranges to block.
Another great attempt early this morning was to add a secondary IP address to the server that was being attacked, and have nFrame put a bandwidth limit on the original IP. Since we own most the domain names for the web sites we host, we could change their domain to point at the alternate IP address. This didn’t work as well as expected however because traffic started to increase on that alternate IP address as well, showing that it was not the IP address of the server being attacked, but instead a particular domain name that resides on the server. The ICMP requests didn’t leave any clues as to which domain was being targeted so it was a guessing game. Since the attack had been going on for more than 24 hours, and was estimated by nFrame employees to have the potential to flood up to 300 mbps, the attack was considered a risk to the entire nFrame facility. Once they got involved, it wasn’t long before they figured out that all of the requests were coming predominately from one ISP. Since most of the requests were coming in the form of blank packets 40 kb in size, they simply dropped all packets 40 kb in size or lower. The sites are still loading a little slow, but this will have to do till the firewall is in place. This was a crazy experience seeing a DDoS firsthand, and I was blown away by the brute force used by the attacker.