*Cyber-rabbit working in the lair. Via* *dreamlike.art*

I recently mentioned that I was getting very little spam on my email server (yes I run my own mail server, and yes I need to publish an update about it) and listed the following things in a summary as to my success: greylisting, DNS blackholes, SPF, DKIM, DMARC, and Pi-hole with Unbound in recursive mode. The latter seemed to have attracted some attention and a few people contacted me directly about it, so I thought I’d expand upon it.

While I am a “home lab person” I should also note for those of you that don’t know, I manage public servers for a hacker group and have done so for a couple of decades. I want to protect the systems with public IP addresses as well as the devices on my internal network with a single Pi-hole instance. To help set the stage, here is a simplified network diagram to illustrate what we’re dealing with:

*A slightly more detailed (and the latest version) of this diagram can be found* *here*.

Why should you care? If you’re a home lab person, a person concerned about privacy, or simply a person that wants to improve your network’s security then this is for you. Even a company can do this - it isn’t limited to a home network. The fact that this setup helped block spam is just another benefit of running Pi-hole as far as I’m concerned.

And what exactly is Pi-hole? It is simply a service that you can install yourself that takes up very little resources, it checks your DNS lookups to see if they are associated with sites that are malicious or want to collect data on you and your Internet traffic, and blocks the bad by not resolving them to their IP address - basically returning info that states they could not be resolved (yes I am simplifying things). The list of sites to block can be extensive, you can set Pi-hole up to subscribe to multiple lists, and it is like a browser adblocker on steroids that works on all traffic. It is called Pi-hole as the original intent was that one could easily run it on a Raspberry Pi, although you don’t have to install it that way. I have a specific server I wanted to install it on that performs a few different functions, and this has become a rather important function that I wanted several public servers to take advantage of, it was installed on Daemon - the old DNS server.

Now when I say “the old DNS server”, I mean because I am now using my registrar Gandi for DNS, mainly to take advantage of simply clicking a button and turning on DNSSEC. Gandi was already handling backup DNS service for the nmrc.org domain with Daemon as the main DNS server (more on that original role here), but now Daemon performs other functions (rsyslog server, Veilid node).

Each of the 5 public IP addresses are individually isolated - strict firewall rules as well as MFA enabled for privileged access. Administrative access is prevented from the outside directly - NMRC members can currently access services they are configured to access with MFA (e.g. Mastodon services on Rigor-Mortis) and using Cloudflare tunneling can get shell access to Talon (using multiple MFA). For ease of implementation, having Pi-hole on Daemon meant I did not have to open up access through Vortex for the other public servers to reach it - I only would have to open up access to the other public IP addresses and that was it.

The Basic Install

Pi-hole documentation can be found here: https://docs.pi-hole.net/. Following the menu on the left side, click on getting started and proceed from there. Daemon runs Ubuntu Server 22.04 system, so I followed those instructions. I took all of the defaults and got a very basic setup installed. As mentioned I had port 53 open to the other servers for DNS, and I added port 80 (for remote Pi-hole management) to allow for access from my internal network only.

I configured my adlists in the interface, and ended up with a total of 18 (yes I am sure there are duplicate entries between lists, but Pi-hole takes care of that). I believe during the initial installation, there is one installed, and I added several more. If you are wondering about them, I simply did a few searches and read reviews of various lists, and started adding them. Here are my current 18:

After initial testing, things seemed to be working okay, so I moved to the next phase.

*As you can see, the 18 lists add up to well over a quarter million unique domains.*

A Bit More Advanced

Unbound was set up and configured using the instructions located here: https://docs.pi-hole.net/guides/dns/unbound/. I made sure regular DNS forwarding was disabled and added a custom DNS via unbound. Running in recursive mode means that I point the systems I want at Daemon for DNS, and if the DNS request is for “google.com” it does not ask my upstream ISP or some other DNS server out there, it goes straight to the “.com” root server and asks there (assuming it’s not already in local cache). This helps eliminate the chances that some type of interception upstream is in place that maliciously redirects NMRC Internet traffic elsewhere.

The instructions were fairly easy to follow, so testing commenced and seemed to work fine. My testing consisted of one laptop connecting up to a test Wi-Fi network on its own VLAN, with that VLAN’s DHCP server handing out IP addresses with Daemon as the DNS server. Slowly, over the course of a few days, I added other VLANs until my entire internal network was using Daemon’s Pi-hole for DNS. After each one, I looked for problems with those devices - fortunately I experienced none.

In the screenshot below, you will notice that there are 5 active clients. One of those is Daemon (listed as localhost), and another is Vortex, which is the UniFi Dream Machine Pro that functions as a gateway between the internal and external networks. Vortex accounts for the vast majority of DNS traffic using Pi-Hole. The remaining 3 are Blackhole, Rigor-mortis, and Talon. These were moved over one at a time and monitored carefully.

Blackhole. This is a GitLab instance, used by NMRC members for coding and other projects. It barely uses DNS and rarely is anything blocked. This one was tested and monitored during a code build for maybe an hour before considering it fine.
Rigor-mortis. This is the NMRC web server as well as the NMRC Mastodon instance for NMRC members. There are parts of its traffic (associated with Mastodon traffic) that is blocked regularly, but these are in fact malicious sites. Usually these are instances associated with spam, hate postings, and bots - these get blocked later on through normal admin procedures but it’s nice to know that things like bots are blocked fairly quickly.
Talon. The NMRC mail server. As I mentioned earlier, this is set up with greylisting, DNS blackholes, SPF, DKIM, and DMARC as well as various levels of filtering (such as a large /etc/mail/access list with 25k lines), and while there is some redundancy between the DNS blackholes I have configured within Sendmail and what Pi-hole provides, it is helping to prevent some of the malicious emails from getting delivered. For example, the week I am typing this the only questionable items I’ve received have originated from Gmail and I’ve only received two of those. Everything else has been blocked. Example of a “block” from Pi-hole where the sender of the spam site does not resolve and is rejected:

Nov 14 11:48:37 talon sm-mta[273071]: 3AEHmZjW273071: ruleset=check_rcpt, arg1=<thegnome@nmrc.org>, relay=mail173.on.mlsend.com [185.249.220.173] (may be forged), reject=451 4.1.8 Domain of sender address 2347375833271769167@mlsend.com does not resolve

*This is for a 24 hour period, the numbers do vary but this is a decent representation.*

Other Notable Updates

Since I was having to change the DNS server on several Linux systems that had static IPs, I took the time to make sure the network interfaces were set up the way I preferred. Using the output from resolvectl status I made a few changes to the interfaces that needed it.

resolvectl llmnr <interface> off (to disable LLMNR)
resolvectl dnssec <interface> on (to enable DNSSEC, although unsure as to the exact impact)
resolvectl dns <interface> 162.196.226.77 (set DNS server to Daemon’s address)
netplan apply (ensure settings survive a reboot)

The cronjob located at /etc/cron.d/pihole was tweaked slightly. Instead of the ad sources being updated once a week, I set them to run daily at 2:00am, and I changed the time for automatic updates to happen early evening so if problems occurred it happens past the normal work day but before various automated updates run overnight - my thinking is that I’d notice something amiss in the evening. This has captured an error exactly once and Pi-hole has been up and running for well over a year (I can’t even recall the error, I just remember that it was resolved quickly, probably with a reboot).

Results

As I noted, this has made a good dent in incoming spam/scam/malware in email. Most of the spam that does make it through is coming from gmail addresses anyway, which I typically forward to abuse@gmail.com for whatever that’s worth.

There are other benefits as well - all web browsers and various phone apps on phones that are attached to internal networks are blocked from most tracking, and all of the various spammy spying crap that shitty IoT devices around the house spew out is blocked. But the reduction in email crap? Icing on the cake.

Are there downsides? Absolutely. Normally in Pi-hole’s web interface, you can click on blocked sites to get more information, however when it simply cites the client as “vortex” I have to use a system of internal sniffing and educated guesses to determine the actual internal client involved. And this isn’t always easy. Spam mail has one problem specifically relating to inbound mail: by generating a “reject 451 4.1.8” on a remote domain’s failure to resolve doesn’t prevent the repeated attempts - this does require manual intervention to shut off the remote site as a 451 error is considered a “temporary” error. I’d have to go in and configure a specific 5xx error to cause the email to fail instead of repeatedly retrying or block the sender’s IP range with firewall rules (or both). But at least I know about it. Pi-hole does make it easier to at least spot these.

In setting things up where I had everything handled from a single Pi-hole solution, it meant that I have just one console I need to view. If I needed to really deep dive and explore those individual internal systems I could set up a second internal Pi-hole, and even point it at the other one for the recursive benefit (just to see if it might work). Ah well, maybe another day, things are currently workable.

Conclusion

Overall though, I am more than pleased. It has cleaned up a lot of traffic, and I feel like from a privacy standpoint this is completely positive. Highly recommended, would rabbit hole again.

I think solutions like this are going to be more and more common, what with growing concerns about things like eIDAS (including EFF’s take on this) and major vendors imposing their will on adblockers in browsers, things like Pi-hole will allow at least some control of ad blocking back in the hands of Internet users. Hopefully this blog post gives you some ideas for taking back a bit more control.

Mark Loveless, aka Simple Nomad, is a researcher and hacker. He frequently speaks at security conferences around the globe, gets quoted in the press, and has a somewhat odd perspective on security in general.