TLS Certificates for Internal Services: Split-Horizon DNS and ACME Implementation

TLS Certificates for Internal Services: Split-Horizon DNS and ACME Implementation

The Challenge of Internal TLS

Securing internal services with TLS often presents a trade-off between user convenience and security. The primary difficulty lies in certificate trust: using self-signed certificates or a private Certificate Authority (CA) requires every client device to manually trust the root certificate, or users must be instructed to ignore browser warnings, which degrades security posture.

The Split-Horizon DNS Approach

Split-horizon DNS solves the trust problem by allowing a single domain name to resolve to different IP addresses depending on the client's location. For public DNS resolvers, a domain (e.g., grafana.tuxnet.dev) resolves to a public IP, enabling the use of public CAs like Let's Encrypt or ZeroSSL for certificate issuance. For clients connected via VPN, the same domain resolves to an internal IP address.

This architecture removes the need for manual certificate installation on client machines while maintaining encryption. To prevent public access to internal services, a Web Access Firewall (WAF) or network binding is used to reject traffic that does not originate from the VPN.

Technical Implementation Guide

Required Tooling

  • VPN with DNS Resolver: NetBird is used to handle the split-horizon DNS logic via Custom Zones.
  • ACME Client: acme.sh is used for automated certificate issuance and renewal.
  • Reverse Proxy: Nginx acts as the entry point and provides the WAF layer.

Certificate Issuance

Using the http-01 challenge, certificates can be issued in standalone mode, meaning Nginx does not need to occupy port 80 permanently:

acme.sh --issue -d grafana.tuxnet.dev --server letsencrypt --standalone

Nginx Configuration for Internal Binding

To ensure the service is only accessible via the VPN, Nginx should bind specifically to the VPN network interface rather than all interfaces:

server {
    listen our-server.netbird.cloud:443 ssl;
    server_name grafana.tuxnet.dev;
    
    ssl_certificate     /etc/ssl/certs/grafana.tuxnet.dev.crt;
    ssl_certificate_key /etc/ssl/private/grafana.tuxnet.dev.key;

    location / {
        proxy_pass http://grafana;
        proxy_set_header Host $host;
    }
}

Automation and Renewal

Automating renewal requires a cron job that triggers acme.sh --cron and subsequently syncs the certificates to the Nginx directory and reloads the service. To run acme.sh as a non-root user while still binding to privileged port 80, the CAP_NET_BIND_SERVICE capability can be applied to socat:

setcap CAP_NET_BIND_SERVICE=+ep /usr/bin/socat1

Scaling with SANs and CNAMEs

To avoid managing separate certificates for every internal service, Subject Alternative Names (SANs) can be used. By creating a single certificate that covers multiple subdomains, one certificate file can be reused across multiple Nginx server blocks.

Example issuance for multiple domains:

acme.sh --issue -d internal.tuxnet.dev -d grafana.tuxnet.dev -d analytics.tuxnet.dev --server letsencrypt --standalone

Alternative Perspectives and Trade-offs

While split-horizon DNS provides a seamless client experience, technical community discussions highlight several critical trade-offs and alternative strategies:

DNS-01 Validation vs. HTTP-01

Many experts suggest using DNS-01 validation instead of HTTP-01. DNS-01 allows certificates to be issued without the service being publicly reachable on port 80, as validation happens via a TXT record in the public DNS. This eliminates the need for a public-facing WAF or temporary port openings.

Privacy and Certificate Transparency (CT) Logs

Publicly issued certificates are recorded in Certificate Transparency logs. This means that any internal subdomain (e.g., secret-project.example.com) becomes public knowledge, even if the IP address is internal. To mitigate this, some users recommend:

  • Wildcard Certificates: Using *.example.com hides specific internal hostnames from CT logs.
  • Internal CA: Using a tool like step-ca to keep all identity and naming data entirely offline.

Architectural Critiques

Some engineers argue that split-horizon DNS introduces "split-brain" behavior that can make debugging difficult, as clients receive different responses based on their network route. Alternatives mentioned include:

  • BeyondCorp Model: Using mTLS (mutual TLS) certificates on trusted devices rather than relying on VPNs.
  • Direct Internal Mapping: Mapping a public domain to an internal IP in public DNS and relying solely on the VPN for access, avoiding the complexity of split-horizon DNS entirely.

Sources