Wildcard SSL dropping on lots of domains, re-adding it fails.

Trying to add Wildcard hostname to an SSL certificate fails everytime. Reset DNS, confirmed the record exists, queried DNS manually to confirm the record exists and no issues. Still adding * fails.

Description

Action failed

Letsencrypt_Module::request(): Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): dns challenge failed: Challenge failed (response: {"type":"dns-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.customer-domain.net - check that a DNS record exists for this domain","status":400},"url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/260241594386\/Czb8jw","token":"gdbAybneVAfT0YNtkX0_aVyekLx8G54oeOvG1SN-ZOE","validated":"2023-09-01T01:41:55Z"}).
Letsencrypt_Module::request(): Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): Failed to finalize certificate request: Certificate request failed (response: The order has not been validated)

Steps to Reproduce

Not sure it can be reproduced elsewhere, but it’s happening for several domains and nothing has changed with the PowerDNS backend, Galera is in sync and records are replicated among servers.

Environment

ApisCP version:

revision: fc84cec46bed587ae33dcba1933882b94cab501c
timestamp: 1692115617
ver_maj: 3
ver_min: 2
ver_patch: 38
ver_pre: ''
dirty: false
debug: false

Operating System: 5.4.251-1.el7.elrepo.x86_64

# dig -t txt @1.1.1.1 _acme-challenge.customer-domain.net

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.14 <<>> -t txt @1.1.1.1 _acme-challenge.customer-domain.net
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29054
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme-challenge.customer-domain.net.  IN      TXT

;; ANSWER SECTION:
_acme-challenge.customer-domain.net. 120 IN    TXT 
"PoipEI2MMydgEmyOhHpagU3FGqpMdff8l23vwsiZTpI"

;; Query time: 22 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Aug 31 21:50:15 EDT 2023
;; MSG SIZE  rcvd: 113

I’m not sure how to further troubleshoot this as it seems like a Let’s Encrypt issue which seems unlikely to go unresolved for so long.
This affects multiple domains on multiple servers with a mix of CentOS 7 and RockyLinux 8.

Few things to check out:

  • Verify the A and AAAA records published for your nameservers are valid and reachable. I’ve seen situations where AAAA records are published for an NS even though the NS isn’t configured to answer queries on AAAA.
  • When using dig to check your nameserver health, always add +norec to prevent recursion. The nameserver must be authoritative for the question.
  • Ensure no additional nameservers are listed for the domain. This can been verified via whois.sc. I’ve also seen situations where a domain has ns1/2.apisnetworks.com + ns501.domaincontrol.com; whenever ns501 is picked up round-robin… it of course fails.
  • Confirm all nameservers are resolving the record correctly by looking up at each ns:
    for ns in ns1 ns2; do 
       dig +short +norec TXT _acme-challenge.domain.com @${ns}.hostingns.com
    done
    
  • Run the request in debug mode. env DEBUG=1 cpcmd -d DOMAIN letsencrypt:append '*.DOMAIN' for additional information that may be relevant.

What’s the PowerDNS version on all nameservers? pdns _server --version

This worked, but through the GUI it failed.
No AAAA records, no extra nameservers defined and no recursion issues with Dig.

Nothing has worked except running via CLI and nothing has changed since I created this thread. I’ll try this again the next time I get a customer report about SSL being dropped. If I had to guess, it’s not updating pdns with the proper token and the query ends up failing.

PowerDNS isn’t local, it’s remote and the remote version is 4.2.3

~# pdns_control version
4.2.3

This has made me realize that PowerDNS hasn’t been updating, so that’s the next task :slight_smile:

1 Like

I’m not sure I see it.

cpcmd -d DOMAIN dns:get-hosting-nameservers DOMAIN

This value should be set in config/auth.yaml to match the hosting nameservers for the domain. For each of these nameservers returned by the API command, dig @NS TXT _acme-challenge.DOMAIN is queried (via Solvers\Dns). This will repeat up to [letsencrypt] => dns_validation_wait, default 30 seconds.

PowerDNS implements a separate cache for 20 seconds configured under pdns.deadline in config/auth.yaml that reflects its packet cache for authoritative responses. 30 seconds > 20 seconds, so that cache shouldn’t be an issue.

The only thing I can think of - if nameservers aren’t incorrectly configured as 127.0.0.1 as reported by dns:get-hosting-nameservers - is record hasn’t yet propagated from master, slave queried, because non-authoritative is permitted the query is forwarded to master as a recursor which reports the correct answer. Let’s Encrypt in turn makes an authoritative request to the slave resulting in an invalid response.

If this update on edge doesn’t fix it, verify the TXT record via unboundtest.com next time you spot it. This configuration is similar to what Let’s Encrypt uses.

Well, still seeing this issue, confirmed TXT record via unbound, also confirmed nameservers and everything else looks good. Anything else I can try?

A full log of running env DEBUG=1 cpcmd -d siteXX letsencrypt:append '*.baddomain.com' would be the next thing to evaluate.

Today it worked, last night it did not. Next time a domain fails I’ll provide a full dump of the debug output as well as the unboundtest.com output.

~]# env DEBUG=1 cpcmd -d site6 letsencrypt:append '*.addondomain.com'
DEBUG  : *.primarydomain.com already resolved by dns
DEBUG  : addondomain.com already resolved by http
DEBUG  : primarydomain.com already resolved by http
DEBUG  : SSL challenge attempt: dns (*.addondomain.com)
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : Setting DNS TXT record _acme-challenge.addondomain.com with value tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 1/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 2/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 3/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 4/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 5/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 6/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 7/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 8/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 9/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 10/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 11/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 12/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 13/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 14/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 15/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 16/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 17/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 18/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 19/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 20/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 21/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 22/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 23/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 24/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 25/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 26/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 27/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 28/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 29/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 30/30
WARNING: Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): dns challenge failed: Challenge failed (response: {"type":"dns-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.addondomain.com - check that a DNS record exists for this domain","status":400},"url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/268773936546\/B9yl2g","token":"SeElImV5XO5zwJ7yaZmp4Az4_bq3jAQ46L1c8XpTwA8","validated":"2023-09-28T13:06:40Z"}).
ERROR  : Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): Failed to finalize certificate request: Certificate request failed (response: The order has not been validated)
         0. Error_Reporter::add_error("Failed to finalize certificate request: %s", ["Certificate request failed (response: The order has not been validated)"])
            [/usr/local/apnscp/lib/log_wrapper.php:62]
         1. error("Failed to finalize certificate request: %s", "Certificate request failed (response: The order has not been validated)")
            [/usr/local/apnscp/lib/Opcenter/Crypto/Letsencrypt/AcmeDispatcher.php:427]
         2. Opcenter\Crypto\Letsencrypt\AcmeDispatcher->solve(AcmePhp\Core\Protocol\CertificateOrder)
            [/usr/local/apnscp/lib/Opcenter/Crypto/Letsencrypt/AcmeDispatcher.php:175]
         3. Opcenter\Crypto\Letsencrypt\AcmeDispatcher->issue(["addondomain.com", "primarydomain.com", "*.addondomain.com", "*.primarydomain.com"])
            [/usr/local/apnscp/lib/Module/Support/Letsencrypt.php:260]
         4. Module\Support\Letsencrypt->requestReal(["addondomain.com", "primarydomain.com", "*.addondomain.com", "*.primarydomain.com"], "site6", true)
            [/usr/local/apnscp/lib/modules/letsencrypt.php:398]
         5. Letsencrypt_Module->request(["addondomain.com", "primarydomain.com", "*.addondomain.com", "*.primarydomain.com"], true, true)
            [/usr/local/apnscp/lib/modules/letsencrypt.php:516]
         6. Letsencrypt_Module->append([*.addondomain.com:0])
            [/usr/local/apnscp/lib/Module/Skeleton/Standard.php:145]
         7. Module\Skeleton\Standard->_invoke("append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/apnscpfunction.php:992]
         8. apnscpFunctionInterceptor->call("letsencrypt_append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/CLI/cmd.php:62]
         9. CLI\__call("letsencrypt_append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/CLI/cmd.php:581]
        10. CLI\main()
            [/usr/local/apnscp/bin/cmd:7]
ERROR  : Letsencrypt_Module::request(): Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): dns challenge failed: Challenge failed (response: {"type":"dns-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.addondomain.com - check that a DNS record exists for this domain","status":400},"url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/268773936546\/B9yl2g","token":"SeElImV5XO5zwJ7yaZmp4Az4_bq3jAQ46L1c8XpTwA8","validated":"2023-09-28T13:06:40Z"}).
         0. Error_Reporter::add_error("Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): dns challenge failed: Challenge failed (response: {"type":"dns-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.addondomain.com - check that a DNS record exists for this domain","status":400},"url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/268773936546\/B9yl2g","token":"SeElImV5XO5zwJ7yaZmp4Az4_bq3jAQ46L1c8XpTwA8","validated":"2023-09-28T13:06:40Z"...")
            [/usr/local/apnscp/lib/Module/Support/Letsencrypt.php:271]
         1. Module\Support\Letsencrypt->requestReal(["addondomain.com", "primarydomain.com", "*.addondomain.com", "*.primarydomain.com"], "site6", true)
            [/usr/local/apnscp/lib/modules/letsencrypt.php:398]
         2. Letsencrypt_Module->request(["addondomain.com", "primarydomain.com", "*.addondomain.com", "*.primarydomain.com"], true, true)
            [/usr/local/apnscp/lib/modules/letsencrypt.php:516]
         3. Letsencrypt_Module->append([*.addondomain.com:0])
            [/usr/local/apnscp/lib/Module/Skeleton/Standard.php:145]
         4. Module\Skeleton\Standard->_invoke("append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/apnscpfunction.php:992]
         5. apnscpFunctionInterceptor->call("letsencrypt_append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/CLI/cmd.php:62]
         6. CLI\__call("letsencrypt_append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/CLI/cmd.php:581]
         7. CLI\main()
            [/usr/local/apnscp/bin/cmd:7]
ERROR  : Letsencrypt_Module::request(): Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): Failed to finalize certificate request: Certificate request failed (response: The order has not been validated)
         0. Error_Reporter::add_error("Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): Failed to finalize certificate request: Certificate request failed (response: The order has not been validated)")
            [/usr/local/apnscp/lib/Module/Support/Letsencrypt.php:271]
         1. Module\Support\Letsencrypt->requestReal(["addondomain.com", "primarydomain.com", "*.addondomain.com", "*.primarydomain.com"], "site6", true)
            [/usr/local/apnscp/lib/modules/letsencrypt.php:398]
         2. Letsencrypt_Module->request(["addondomain.com", "primarydomain.com", "*.addondomain.com", "*.primarydomain.com"], true, true)
            [/usr/local/apnscp/lib/modules/letsencrypt.php:516]
         3. Letsencrypt_Module->append([*.addondomain.com:0])
            [/usr/local/apnscp/lib/Module/Skeleton/Standard.php:145]
         4. Module\Skeleton\Standard->_invoke("append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/apnscpfunction.php:992]
         5. apnscpFunctionInterceptor->call("letsencrypt_append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/CLI/cmd.php:62]
         6. CLI\__call("letsencrypt_append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/CLI/cmd.php:581]
         7. CLI\main()
            [/usr/local/apnscp/bin/cmd:7]
----------------------------------------
MESSAGE SUMMARY
Reporter level: ERROR
ERROR: Letsencrypt_Module::request(): Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): dns challenge failed: Challenge failed (response: {"type":"dns-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.addondomain.com - check that a DNS record exists for this domain","status":400},"url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/268773936546\/B9yl2g","token":"SeElImV5XO5zwJ7yaZmp4Az4_bq3jAQ46L1c8XpTwA8","validated":"2023-09-28T13:06:40Z"}).
ERROR: Letsencrypt_Module::request(): Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): Failed to finalize certificate request: Certificate request failed (response: The order has not been validated)
----------------------------------------

Unbound:

Query results for TXT addondomain.com

Response:
;; opcode: QUERY, status: NOERROR, id: 48867
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: do; udp: 512

;; QUESTION SECTION:
;addondomain.com.	IN	 TXT

;; ANSWER SECTION:
addondomain.com.	0	IN	TXT	"v=spf1 a mx include:relay.mailchannels.net ?all"

----- Unbound logs -----
Sep 28 13:13:16 unbound[1516568:0] notice: init module 0: validator
Sep 28 13:13:16 unbound[1516568:0] notice: init module 1: iterator
Sep 28 13:13:16 unbound[1516568:0] info: start of service (unbound 1.16.3).
Sep 28 13:13:17 unbound[1516568:0] info: 127.0.0.1 addondomain.com. TXT IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving addondomain.com. TXT IN
Sep 28 13:13:17 unbound[1516568:0] info: priming . IN NS
Sep 28 13:13:17 unbound[1516568:0] info: response for . NS IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 192.58.128.30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: priming successful for . NS IN
Sep 28 13:13:17 unbound[1516568:0] info: response for addondomain.com. TXT IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 2001:500:9f::42#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for addondomain.com. TXT IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <com.> 2001:501:b1f9::30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: resolving ns1.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving ns2.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving ns2.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving ns1.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: response for ns2.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 2001:500:a8::e#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for ns1.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 2001:500:a8::e#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for ns1.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 2001:dc3::35#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for ns2.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <net.> 2001:503:a83e::2:30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for ns1.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <net.> 192.12.94.30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for ns1.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <net.> 2001:500:d937::30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for ns2.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <myns.net.> 104.238.162.256#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: response for ns1.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <myns.net.> 104.238.162.256#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: response for ns1.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <myns.net.> 45.77.110.256#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: response for ns2.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 2001:500:12::d0d#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for addondomain.com. TXT IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <addondomain.com.> 45.77.110.256#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: prime trust anchor
Sep 28 13:13:17 unbound[1516568:0] info: generate keytag query _ta-4f66. NULL IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving . DNSKEY IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving _ta-4f66. NULL IN
Sep 28 13:13:17 unbound[1516568:0] info: response for ns1.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <myns.net.> 104.238.162.256#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was nodata ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: response for _ta-4f66. NULL IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 198.41.0.4#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was NXDOMAIN ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: response for . DNSKEY IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 2001:503:c27::2:30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: validate keys with anchor(DS): sec_status_secure
Sep 28 13:13:17 unbound[1516568:0] info: Successfully primed trust anchor . DNSKEY IN
Sep 28 13:13:17 unbound[1516568:0] info: validated DS com. DS IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving com. DNSKEY IN
Sep 28 13:13:17 unbound[1516568:0] info: response for ns2.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <net.> 192.54.112.30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: resolving ns1.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: response for com. DNSKEY IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <com.> 2001:503:eea3::30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: validated DNSKEY com. DNSKEY IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving addondomain.com. DS IN
Sep 28 13:13:17 unbound[1516568:0] info: response for addondomain.com. DS IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <com.> 192.35.51.30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was nodata ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: NSEC3s for the referral proved no DS.
Sep 28 13:13:17 unbound[1516568:0] info: Verified that unsigned response is INSECURE```

I was able to add wildcard for the primary domain, but not the addon domain.

You’d want to check TXT for _acme-challenge.addondomain.com, not addondomain.com.

This keeps happening on multiple servers.
Everything that I can see checks out. TXT record is created and confirmed.
I’m not sure what to do, customers are getting frustrated and I don’t have a solution for them but to keep trying.

I understand this can be a very difficult time for you and your customers. I still need that unbound log against _acme-challenge.DOMAIN to have a better picture of what I am looking at. Nothing will be actionable until I have the information in front of me.

I know from what I run - BIND - renewals work fine unless Let’s Encrypt fails a CAA lookup using UDP. This may or may not be a contributing cause; it is impossible to know without the requisite diagnostics.

If this update on edge doesn’t fix it, verify the TXT record via unboundtest.com next time you spot it. This configuration is similar to what Let’s Encrypt uses.

env DEBUG=1 cpcmd -d DOMAIN letsencrypt:append ‘*.DOMAIN’

If this fails through the GUI, provide the GUI postback results. Again, to reiterate: I cannot proceed forward until I have more information.