Information Security Asked by Alexis Wilke on September 1, 2020
I wake up this morning to a rebooted server. The DNS server was running at over 100%. After a little bit of work, I got fail2ban in place to block all of those requests.
The requests themselves are valid, jut repeated hundred of times per seconds. Once the block got many (hundred) of IPs, I can see that I am blocking 1 million UDP hits every few hours.
Is that just a [D]DoS attack? (probably considered dynamic since many computers are involved and once one was blocked long enough it looks like it stops the requests)
The one other possibility I can think of is that the attacker is trying to crash the DNS server and gain access when it restarts or crash the whole computer and attempt connections to other services. (i.e. in case you don’t know how to get your firewall in place before you start your services)
Since my last firewall reset, here are my stats:
Hits: 2,346,742
Number of IPs: 473
It goes fast. Several hundred hits per second. The number of IPs doesn’t grow much, however.
From @Schroeder's comment, I made some additional searches and found out much more about this type of attack.
First of all, it looks like I am the target of the attack. Why? Because the DNS Amplification attack means that the source port of the DNS requests is set to 53. This makes it possible to force my DNS to send a non-requested response to another server (see graph below). Since I have less than 0.1% of such hits using that port, I don't really think that my DNS was used for the Amplification, but rather that it was the target.
The attacker sets up a computer with software sending UDP packets to some random DNS servers asking for a domain name. The packet looks normal except for the fact that the attacker puts the IP address of a different computer instead of his IP address as the source. The result is that the DNS sends the reply to the wrong computer:
10.0.0.1 10.0.0.2 10.0.0.3
+------------+ +------------+ +------------+
| | | | | |
| Attacker +-->| Some DNS +-->| Me |
| | | | | |
+------------+ +--+---------+ +------------+
| ^
v |
+---------+--+ This step is not required, it happens if
| | your DNS is setup to accept recursive
| Master DNS | requests (which is not a good idea)
| |
+------------+
10.0.0.4
In the example above, the attacker's request should have 10.0.0.1 as the UDP origination address. But instead, the attacker changes his IP address with 10.0.0.3. As a result, when the DNS at 10.0.0.2 receives the UDP packet and is ready to send a reply, it sends the data to 10.0.0.3.
The Master DNS of the domain is going to be used if the DNS at 10.0.0.2 doesn't know anything about the domain name being queried.
IMPORTANT NOTE: This is true of all UDP services. DNS is probably the worst in terms of amplification, but NTP, for example, can also be targeted.
It's not possible from 10.0.0.2 because UDP has no recollection of the real source of the packet.
What is possible is for ISPs to verify that all the packets going out of a given computer has the IP address of said computer. Some do, I think, but most probably do not. It has a small speed impact... which of course with a DNS amplification is to laugh at (the DNS amplification has a way worse effect on the Internet than a small check of UDP packet origins).
The other thing may be that a user may still be able to do by connecting to the Internet in such a way that he bypasses the ISP verification. I do not know whether and/or how that would be possible, but I wouldn't be surprised that it could be done.
Actually, this is very problematic since the origin of such attacks is difficult to track and that's certainly why they are still occurring en masse.
The packet to request a DNS record, which is what happens here, is very small, maybe 300 bytes. So the attacker only has to send one small UDP packet.
The response, however, may be multiple kilobytes of data. Specifically, there is a domain which is being used in these attacks:
peacecorps.gov
and that domain returns over 3Kb of data! (It defines many 'TXT' fields which are used in that amplification attack.) This is why it is called Amplification Attack.
As a result, the requests from the Attacker are transformed in about 10× bigger responses and the computer being attacked (10.0.0.3) gets swamped with a large number of rather large UDP packets.
Here I show the resulting entry in iptables
. The first number represents the number of hits to my DNS computer after about 40 minutes (so over 10,000 hits a minute...):
pkts bytes target prot opt in out source destination
61637 4376227 DROP udp -- eno1 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 STRING match "|0a7065616365636f72707303676f76|" ALGO name bm TO 65535
Notice also how the string was fully transformed to hexadecimal numbers.
HTTP uses TCP which is a two way protocol. So no amplification can be generated with TCP. i.e. TCP breaks if you did not first properly connect with the full handshake.
Actually, some people are talking about using HTTP to query for domain names. (DoH — Domain over HTTP).
The amplification happens because your DNS is setup to accept requests of domain names that you do not control (i.e. domains you're not the owner of).
BIND has an option that allows it to do amplification, which is called recursion in DNS parlance. This feature is now off by default, but you may have turned it on (by mistake?).
Here is the wrong setting:
allow-recursion { any; };
What you want is to use the correct settings:
trusted-servers { 192.0.2.4; } // list all your trusted servers
allow-recursion { trusted-servers; };
This will make your server automatically deny all such requests. So if you are not peacecorps.gov
and you receive such a request, BIND will just stop right there and write a note in your DNS logs about the denied request.
Yes. I started by doing that because my server was running well over 100% in CPU time. However, it may not be wise to do so. From the picture above, you can see that your DNS server sits between an attacker and a victim. If you block the source IP address, it's not the attacker's IP that you are blocking, it's the victim's. This means you're actually preventing the victim from seeing your domain names and do legitimate requests. That's probably not what you want!
At first, I generated a log message from my firewall. If I were to detect 5 or more requests to port 53 (UDP) from the same IP address in a short amount of time (5 sec.), then I would block the IP address. To do so, I used fail2ban
.
First, I have a filter which detects the links with UDP and port 53 as the destination:
# Filter: /etc/fail2ban/filter.d/named-fast-requests.conf
[Definition]
failregex = sIN=[a-z0-9]+ .* SRC=<HOST> .* PROTO=UDP .* DPT=53s
Second, I have a jail which gives the other parameters:
# Jail: /etc/fail2ban/jail.d/named-fast-requests.conf
[named-fast-requests]
enabled = true
filter = named-fast-requests
action = named-action[scheme=all,period=year,reason=named fast requests]
logpath = /var/log/iptables/iptables.log
maxretry = 5
findtime = 5
bantime = 1036800
The main point here is the maxretry
and findtime
which are set at 5 times in 5 seconds or less. When that happens, I block the <HOST>
.
You will want to update the action with your own thing. I use my own iplock
tool here. The command I use in the action:
actionban = /usr/sbin/iplock -s named -b <ip>
By default, fail2ban uses iptables
directly. Search their actions to see how it's done.
You should block requests where the source UDP port is less than 1024. These are not valid. If you offer a server, port 53 will be a destination port, not source.
iptables -I INPUT 123 -i eth0 -p udp -m udp --sport 0:1023 -j DROP
WARNING: replace the position (123) with the correct number!
This rule says that for any incoming UDP packet on eth0
, which has a source port between 0 and 1023, drop the packet. This prevents someone from using your DNS server for amplification. However, it does not protect your own server from amplification from others. Note that even with the proper allow-recursion
setup, you would not prevent an amplification attack if the attacker properly selects one of your domain names for the attack.
Obviously, having BIND accept all those requests with peacecorps.gov
is not something anyone wants. You can actually block those directly in your firewall. This works because UDP packets are not encrypted so the domain name is visible.
Here is a rule one can use to block those domain name requests:
sudo iptables -I INPUT 123 -i eno1 -p udp -m udp --dport 53
-m string --hex-string "|0A|peacecorps|03|gov|" --algo bm -j DROP
Obviously, if your DNS has any domain or sub-domain including "peacecorps.gov" then it should not use that iptables
rule. For most of us, though it should be rare.
The --hex-string
option allows you to specify a string. The way it is defined in the UDP packet uses a form of P-string (size + data). Since "peacecorps" is 10 character long, we put 0x0A just before it. Again, the "gov" is three letters so we use 0x03. If we were to use the string "peacecorps.gov", it wouldn't work since the period would not match the 0x03 byte. The first size is optional, though (although you'd match anything that looks the same such as "bestpeacecorps").
Having such a rule will save your domain name service a ton of totally unwanted traffic.
Source: https://defragged.org/2020/05/20/tips-and-tricks-blocking-dns-requests-via-iptables/
In order to see what query your DNS server receives and replies to, you can run the following commands in your console:
sudo rndc querylog
Now look at the logs, usually here:
less /var/log/named.log
Look at the bottom (hit G in less
) and you should start seeing queries from remote IPs. The logs include the domain name being checked. It's very practical, especially if you missed entering some of your own domain names in your secondary or tertiary DNS.
Correct answer by Alexis Wilke on September 1, 2020
Get help from others!
Recent Questions
Recent Answers
© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP