When cert-manager Says Your DNS-01 Record Hasn't Propagated (But It Has)

I wanted real Let’s Encrypt certificates for a couple of services on my k3s homelab — cms.clefsoft.co.uk and a few others. They’re LAN-only: nothing’s exposed to the internet, so HTTP-01 was a non-starter (Let’s Encrypt can’t reach an ingress that doesn’t answer on the public internet). That leaves DNS-01 — prove you control the domain by writing a TXT record — which I wired up with cert-manager and Cloudflare, where the clefsoft.co.uk zone lives.

It created the TXT record instantly. Then it sat there. For an hour. “Not yet propagated.” Except it had.

The setup, which is textbook

A ClusterIssuer using the ACME DNS-01 solver, a Cloudflare API token scoped to Zone:DNS:Edit in a secret, the whole thing following the docs to the letter:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

Request a certificate and the order gets stuck on the challenge:

$ kubectl describe challenge cms-clefsoft-xxxx
Status:  pending
Reason:  Waiting for DNS-01 challenge propagation: DNS record for
         "cms.clefsoft.co.uk" not yet propagated

Red herring: Windows nslookup lies about underscore records

First instinct: is the record actually there? So I checked from my Windows box:

> nslookup -type=TXT _acme-challenge.cms.clefsoft.co.uk

…and got results confusing enough to send me down a rabbit hole about whether cert-manager was even writing the record. Don’t trust Windows nslookup for _acme-challenge lookups. The underscore-prefixed name trips it up, and it’ll happily give you a misleading answer.

Check it a way that can’t lie to you — a DNS-over-HTTPS query straight to a public resolver:

curl -s 'https://1.1.1.1/dns-query?name=_acme-challenge.cms.clefsoft.co.uk&type=TXT' \
  -H 'accept: application/dns-json' | jq '.Answer'

The record was there, live, correct, visible to the entire internet. Cloudflare had it. Let’s Encrypt could see it. cert-manager was the only thing in the world that claimed it wasn’t propagated.

The actual cause: it’s asking the wrong resolver

Here’s the bit the “not yet propagated” message does a terrible job of telling you. Before cert-manager asks Let’s Encrypt to validate, it runs its own propagation self-check — it looks up the TXT record itself and waits until it can see it. And that self-check resolves DNS through the cluster’s normal path: the pod’s resolver → CoreDNS → whatever upstream CoreDNS forwards to.

On this network, that upstream is my local DNS server — the one I run so that cms.clefsoft.co.uk resolves to the node’s LAN address (192.168.1.40) instead of going out to the internet. Which is exactly the problem. That local server is split-horizon for clefsoft.co.uk: it answers for the names I’ve defined internally and has no idea about a _acme-challenge TXT record sitting in Cloudflare’s public zone. So cert-manager asks my local resolver “can you see this TXT record?”, my resolver says “never heard of it”, and cert-manager concludes it hasn’t propagated — forever.

Let’s Encrypt validates against public DNS. cert-manager was validating against my private DNS. They were never going to agree.

The fix: force the self-check onto public resolvers

cert-manager has flags for precisely this. Tell it to do its DNS-01 self-check against specific public recursive nameservers, and only those, bypassing the cluster path entirely. On the cert-manager controller deployment:

args:
  - --dns01-recursive-nameservers-only=true
  - --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53

Now the self-check asks Cloudflare and Google directly — the same public view Let’s Encrypt has — sees the record immediately, and the challenge goes through.

The second bug: two cert-managers fighting

Except it still worked only sometimes. A restart, and it’d be back to “not propagated.”

The cause was entirely self-inflicted, which is the most homelab thing imaginable: earlier trial-and-error had left me with two cert-manager installations in the cluster. Only one had the recursive-nameserver args. They were both running, both contending for the same leader election — and whenever the unpatched one won the lease on a restart, the self-check went back to using the cluster resolver and failed again. Intermittent, maddening, and nothing to do with DNS at all by that point.

The fix was to stop the duplicate contending:

kubectl -n cert-manager scale deploy/cert-manager --replicas=0

leaving the correctly-configured controller as the sole leader. Certificate issued, cms-clefsoft-tls went Ready, and it’s been quiet since.

What I’d tell you before you start

• “Not yet propagated” usually means “the resolver I’m using can’t see it,” not “the record isn’t there.” Verify the record with DoH or dig against a public resolver first — if it’s live publicly, cert-manager is looking in the wrong place.
• If you run split-horizon DNS — and any homelab that points public hostnames at private IPs does — you almost certainly need --dns01-recursive-nameservers-only=true with public nameservers. This is the single config line that fixes the whole class of problem.
• Windows nslookup and _acme-challenge don’t get along. Use DoH or a Linux dig.
• Run one cert-manager. Two installs fighting a leader lease will give you failures that look random and have nothing to do with your actual config.

The through-line — same as every good homelab bug — is that the tool was confidently reporting the wrong thing. cert-manager wasn’t wrong that its lookup failed; it was wrong to call that “not propagated,” because the record had propagated fine to everyone who mattered. It just wasn’t asking them.