Running a private DNS resolver with CoreDNS, podman and Ansible on CentOS 8

published on Wed, 12 Aug 2020

Running a private DNS resolver is useful in quite a few situations, for example in a home lab or on an internal company network; basically everywhere where you want to give names to private systems. CoreDNS is a simple DNS server that can be used in such a situation - if the name rings a bell with you, it's because CoreDNS is the standard in-cluster DNS solution for Kubernetes for some time now. In practice, this means that large microservice architectures rely on it to resolve internal and external hostnames.

I decided to go with CoreDNS for my setup because of that fact (it's a tool proved to be battle-tested in Kubernetes environments I worked with) and the simplicity of configuration - it needs a single configuration file, called Corefile, that defines the DNS resolver's behavior. Moreover, CoreDNS is based on a plugin architecture, which potentially allows to extend it with custom functionality.

The small network that required a private DNS resolver in my case was however too small to run Kubernetes, so it was necessary to find a different way of running it. Since it is a Go program, downloading and running a binary would probably work - But I wanted to run it in a container for better isolation. CentOS 8 added support for a new container runtime largely spearheaded by Red Hat and Fedora called podman. podman is interesting because it gets rid of the omniscient (Docker) daemon, allowing for running containers in a more stand-alone manner. It also supports running containers as non-root users, albeit we won't use this feature for now.

To deploy CoreDNS and maintain its configuration, I chose Ansible. Again, mostly because I worked with it before and because all I need to get started is a SSH key on the target system. The following steps will assume that you already have a running CentOS 8 system (likely a VM) running somewhere in your private network and that you have access to it. I won't go into the basics of Ansible and I recommend you to search for tutorials / getting started guides on it before tagging along.

At the end of this, we will have a private DNS resolver that is capable of resolving public names (with some filtering in place) and resolving one or more private domains we can adjust to our needs.

Important: You should make sure that port 53 (DNS) of your system is not available on the public internet, securing access with firewall or security group rules.

Required Packages 

Depending on the disk/CD image you used to install CentOS 8, not all required tools will be available right away. Besides the container runtime we want to use (podman), let's install two more packages: udica and bind-utils. bind-utils is just useful as it contains the dig command, something we can use to query our DNS resolver later on.

udica is more interesting: It hasn't been mentioned yet, but I would like to integrate with existing security mechanisms on CentOS as much as possible. This means that we will keep SELinux in enforcing mode. udica will enable us to generate SELinux modules that will allow running CoreDNS in a podman container while keeping SELinux up.

All things considered, translated to a task in Ansible this would mean adding this to your coredns role:

- name: install base packages
      - podman
      - udica
      - bind-utils
    state: present

Later on, I will not present tasks for trivial operations in Ansible - I strongly recommend to adapt my findings to your needs.

Setting up a Corefile 

Before we get too much into the details of deploying CoreDNS in podman, we should focus on defining our requirements for the DNS resolver and templating a Corefile around that.

We need to ensure that on the target system, there is a directory we can put our configuration into. My Ansible role creates /etc/coredns for that purpose:

- name: add directory for CoreDNS configuration
    path: /etc/coredns
    state: directory
    mode: '0755'

The bad news first: CoreDNS does not come with any kind of built in ad block solution. A lot of people are running services like Pi-hole on their private network and our solution won't get as sophisticated as that. The good news is, if you want to block nepharious websites in your DNS resolver, you can still do that. Earlier we defined the requirement for our resolver to resolve all kinds of names, so let's look into public names first.

Resolve Public DNS, with a Blocklist 

To implement some kind of blocklist for bad DNS names, we can use the hosts plugin of CoreDNS. It reads the content of a /etc/hosts-style file and serves requests based on that. I have included a list from StevenBlack/hosts as a blocklist that will resolve the hosts on that list to, thus failing resolution. Download one of the lists (depending on what you would like to block) into your Ansible role and copy them to your target system.

Let's set up the part in our Corefile that will allow our resolver to resolve public names with the restrictions of our blocklist in place. To do so, we can ask CoreDNS to look into our blocklist and fallthrough to the next lookup method if a name is not on our naughty list. We will rely on /etc/resolv.conf for upstream DNS, but you can adjust the forward plugin configuration we are including to suit your network situation.

. {
    hosts /etc/coredns/blocklist.hosts {
    forward . /etc/resolv.conf

This short configuration snippet instructs CoreDNS in the following ways:

Resolve Private DNS 

Now that we cover public DNS, let's look into setting up one or more private DNS domains on this resolver. I am using the hosts plugin again to resolve a templated list of DNS names; you can adjust that to your needs and situation.

First, let's generate another /etc/hosts-style file for our private zone. The minimal viable solution for this would be this Jinja2 template:

{% for entry in coredns_entries %}
{{ entry.ip }}  {{ }}
{% endfor %}

This requires passing the host variable coredns_entries to our target host in Ansible - A simple coredns_entries configuration could look like this (let's throw in a variable for our private DNS zone for good measure):

coredns_internal_domain: privatezone.internal
    - ip:
      host: gateway.privatezone.internal
    - ip:
      host: gitlab.privatezone.internal

You can get smarter with this (e.g. using the coredns_internal_domain variable to eliminate the requirement for adding a fqdn each time), but this is the bare minimum that will get us started. If it's possible, consider pulling this kind of information out of your inventory. Ansible should template this to the target system, e.g. into /etc/coredns/internal.hosts. Afterwards, adjust the Corefile (that should become a template at this point) to load our internal hosts file:

{{ coredns_internal_domain }} {
    hosts /etc/coredns/internal.hosts

. {
    hosts /etc/coredns/blocklist.hosts {
    forward . /etc/resolv.conf

There we go! It's important to load your private zones before the big catch-all . block, so CoreDNS won't try to send your private hostnames to public DNS resolvers. Now we have a minimal Corefile on our target system. Let's look into running CoreDNS as a podman container next.

Running CoreDNS in Podman 

The next thing we want to do is figuring out the correct podman command we want to run to start our CoreDNS container. While there is an equivalent to docker-compose available, let's stay with the (mostly docker-compatible) podman cli. We will later wrap this call in systemd to leverage it as supervisor for our container.

The full podman run call that I came up with is this one:

/usr/bin/podman run --name coredns --read-only -p -p -v /etc/coredns:/etc/coredns:ro --cap-drop ALL --cap-add NET_BIND_SERVICE coredns/coredns:1.7.0 -conf /etc/coredns/Corefile

Let's look at the parameters passed here to understand what is going on:

All in all, this tries to limit the application running in the container in what it is capable of doing. If you have any improvements to suggest for this, please reach out on Twitter as I would love to hear about them.

Now, if you try to run this command, it will hopefully fail - CoreDNS will be unable to bind to port 53 (if it does not fail or complain in the logs, you probably disabled SELinux previously - shame on you!). SELinux does not allow it yet. Let's fix that.

Generating a SELinux Policy 

Remember when we installed udica in the beginning? Now its time to shine has come! udica is a small but awesome tool that allows generating SELinux policies from container definitions. It takes away a lot of pain with containers and active SELinux. With that being said, I urge you to read up on SELinux and how it works, giving you the opportunity to further slim down policies if possible.

Let's pull the definition of our “coredns” container, put it into a JSON file and pass it to udica:

$ podman inspect coredns > container.json
$ udica -j container.json coredns

Policy coredns created!

Please load these modules using: 
# semodule -i coredns.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}

Restart the container with: "--security-opt label=type:coredns.process" parameter

This generates a coredns.cil file in the current directory. For me, it looked like this:

(block coredns
    (blockinherit container)
    (blockinherit restricted_net_container)
    (allow process process ( capability ( net_bind_service )))

    (allow process dns_port_t ( tcp_socket (  name_bind )))
    (allow process dns_port_t ( udp_socket (  name_bind )))
    (allow process etc_t ( dir ( getattr search open read lock ioctl )))
    (allow process etc_t ( file ( getattr read ioctl lock open  )))
    (allow process etc_t ( sock_file ( getattr read open  )))

It's relatively small and quite declarative, so the main takeaways from this policy should be:

You could further lock down this policy with your own SELinux labels on /etc/coredns, but for now, this should suffice. It will get us past SELinux as a gatekeeper and improve our security posture because we didn't disable SELinux (yay). To enable this policy, run the command in the udica output.

Interlude: SELinux Policies and Ansible 

As all configuration steps, this should be included in our Ansible role to make the installation reproducable on fresh systems. Unfortunately, Ansible doesn't seem to offer a module that will allow applying this, which means we need to fall back to using the shell module. This is how my role handles applying this:

# tasks/main.yml
- name: create /etc/udica for custom SELinux policies
    path: /etc/udica
    state: directory
    mode: '0755'

- name: create coredns SELinux policy file
    src: files/coredns.cil
    dest: /etc/udica/coredns.cil
    - load coredns SELinux module

# handlers/main.yml
- name: load coredns SELinux module
  shell: 'semodule -i /etc/udica/coredns.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}'

This creates a handler running semodule which is being called upon changes to the SELinux policy file. If you know of a better way to do this, please reach out via Twitter!

systemd Service 

We're almost done! At this point, our CoreDNS container should run successfully and you should be able to resolve hosts with dig @ (replace with your own private IP) as long as you're running the podman command from earlier.

But as mentioned, podman doesn't come with a supervising daemon - Which means that noone will restart the container once it's dead (for whatever reason); Nor will it be restarted upon reboot.

To fix this, let's use something on our system that is already working as a supervisor to services: systemd! Fortunately, integration with systemd works quite well under podman, as long as you follow a template recommended by Red Hat.

This is the Jinja template that I use to generate /etc/systemd/system/coredns.service, which is based on the service unit template linked above. The main difference to a standard systemd service unit is that podman will write to PID files that systemd will read to be informed about the container status.

Description=CoreDNS private DNS in a container

ExecStartPre=/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
ExecStartPre=-/usr/bin/podman rm -f coredns
ExecStart=/usr/bin/podman run --name coredns --read-only --security-opt label=type:coredns.process -p {{ ansible_eth1.ipv4.address }}:53:53/tcp -p {{ ansible_eth1.ipv4.address }}:53:53/udp -v /etc/coredns:/etc/coredns:ro --cap-drop ALL --cap-add NET_BIND_SERVICE --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid -d coredns/coredns:1.7.0 -conf /etc/coredns/Corefile
ExecStop=/usr/bin/podman stop coredns


Beyond what the template requires, note the additions in comparison to the previous iteration:

This will deliver a fully functional systemd unit called coredns.service that will utilise podman and a targeted SELinux policy to run CoreDNS on your local network. Do not forget to write your Ansible role in a way that changes to the SELinux policy or your configuration files will trigger a restart.

Wrapping Up 

That's it - We now have CoreDNS running, savely wrapped into podman, SELinux and systemd. Using Ansible is a nice bonus, allowing us to deploy our private DNS resolver to additional systems with little overhead. Not all steps have been presented as Ansible snippets, but I hope it will prove useful to those writing better roles than I do.

If everything worked well, we will now see a container running in podman:

$ podman ps
CONTAINER ID  IMAGE                            COMMAND               CREATED       STATUS           PORTS                                         NAMES
16f66c7140ed  -conf /etc/coredn...  2 hours ago  Up 2 hours ago>53/tcp,>53/udp  coredns

To access the logs (which includes all requests the server received and answered), we can run podman logs coredns.

This setup obviously has further room for improvement. Next steps to explore could be (but are not limited to):

If you have any questions on this or if you would like to point any mistakes or improvements to this - again - please reach out on Twitter via @MarvinBeckers. I'm very happy about any feedback on this to incorporate into further blog posts.