Wildcard SSL dropping on lots of domains, re-adding it fails.

Troy · September 1, 2023, 1:54am

Trying to add Wildcard hostname to an SSL certificate fails everytime. Reset DNS, confirmed the record exists, queried DNS manually to confirm the record exists and no issues. Still adding * fails.

Description

Action failed

Letsencrypt_Module::request(): Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): dns challenge failed: Challenge failed (response: {"type":"dns-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.customer-domain.net - check that a DNS record exists for this domain","status":400},"url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/260241594386\/Czb8jw","token":"gdbAybneVAfT0YNtkX0_aVyekLx8G54oeOvG1SN-ZOE","validated":"2023-09-01T01:41:55Z"}).
Letsencrypt_Module::request(): Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): Failed to finalize certificate request: Certificate request failed (response: The order has not been validated)

Steps to Reproduce

Not sure it can be reproduced elsewhere, but it’s happening for several domains and nothing has changed with the PowerDNS backend, Galera is in sync and records are replicated among servers.

Environment

ApisCP version:

revision: fc84cec46bed587ae33dcba1933882b94cab501c
timestamp: 1692115617
ver_maj: 3
ver_min: 2
ver_patch: 38
ver_pre: ''
dirty: false
debug: false

Operating System: 5.4.251-1.el7.elrepo.x86_64

# dig -t txt @1.1.1.1 _acme-challenge.customer-domain.net

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.14 <<>> -t txt @1.1.1.1 _acme-challenge.customer-domain.net
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29054
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme-challenge.customer-domain.net.  IN      TXT

;; ANSWER SECTION:
_acme-challenge.customer-domain.net. 120 IN    TXT 
"PoipEI2MMydgEmyOhHpagU3FGqpMdff8l23vwsiZTpI"

;; Query time: 22 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Aug 31 21:50:15 EDT 2023
;; MSG SIZE  rcvd: 113

I’m not sure how to further troubleshoot this as it seems like a Let’s Encrypt issue which seems unlikely to go unresolved for so long.
This affects multiple domains on multiple servers with a mix of CentOS 7 and RockyLinux 8.

msaladna · September 3, 2023, 4:15pm

Few things to check out:

Verify the A and AAAA records published for your nameservers are valid and reachable. I’ve seen situations where AAAA records are published for an NS even though the NS isn’t configured to answer queries on AAAA.
When using dig to check your nameserver health, always add +norec to prevent recursion. The nameserver must be authoritative for the question.
Ensure no additional nameservers are listed for the domain. This can been verified via whois.sc. I’ve also seen situations where a domain has ns1/2.apisnetworks.com + ns501.domaincontrol.com; whenever ns501 is picked up round-robin… it of course fails.

Confirm all nameservers are resolving the record correctly by looking up at each ns:

for ns in ns1 ns2; do 
   dig +short +norec TXT _acme-challenge.domain.com @${ns}.hostingns.com
done

Run the request in debug mode. env DEBUG=1 cpcmd -d DOMAIN letsencrypt:append '*.DOMAIN' for additional information that may be relevant.

What’s the PowerDNS version on all nameservers? pdns _server --version

Troy · September 4, 2023, 12:17am

This worked, but through the GUI it failed.
No AAAA records, no extra nameservers defined and no recursion issues with Dig.

Nothing has worked except running via CLI and nothing has changed since I created this thread. I’ll try this again the next time I get a customer report about SSL being dropped. If I had to guess, it’s not updating pdns with the proper token and the query ends up failing.

PowerDNS isn’t local, it’s remote and the remote version is 4.2.3

~# pdns_control version
4.2.3

Troy · September 4, 2023, 12:19am

This has made me realize that PowerDNS hasn’t been updating, so that’s the next task

msaladna · September 10, 2023, 2:22am

I’m not sure I see it.

cpcmd -d DOMAIN dns:get-hosting-nameservers DOMAIN

This value should be set in config/auth.yaml to match the hosting nameservers for the domain. For each of these nameservers returned by the API command, dig @NS TXT _acme-challenge.DOMAIN is queried (via Solvers\Dns). This will repeat up to [letsencrypt] => dns_validation_wait, default 30 seconds.

PowerDNS implements a separate cache for 20 seconds configured under pdns.deadline in config/auth.yaml that reflects its packet cache for authoritative responses. 30 seconds > 20 seconds, so that cache shouldn’t be an issue.

The only thing I can think of - if nameservers aren’t incorrectly configured as 127.0.0.1 as reported by dns:get-hosting-nameservers - is record hasn’t yet propagated from master, slave queried, because non-authoritative is permitted the query is forwarded to master as a recursor which reports the correct answer. Let’s Encrypt in turn makes an authoritative request to the slave resulting in an invalid response.

If this update on edge doesn’t fix it, verify the TXT record via unboundtest.com next time you spot it. This configuration is similar to what Let’s Encrypt uses.

Troy · September 25, 2023, 12:43am

Well, still seeing this issue, confirmed TXT record via unbound, also confirmed nameservers and everything else looks good. Anything else I can try?

msaladna · September 25, 2023, 11:26am

A full log of running env DEBUG=1 cpcmd -d siteXX letsencrypt:append '*.baddomain.com' would be the next thing to evaluate.

Troy · September 25, 2023, 12:26pm

Today it worked, last night it did not. Next time a domain fails I’ll provide a full dump of the debug output as well as the unboundtest.com output.

Troy · September 28, 2023, 1:22pm

~]# env DEBUG=1 cpcmd -d site6 letsencrypt:append '*.addondomain.com'
DEBUG  : *.primarydomain.com already resolved by dns
DEBUG  : addondomain.com already resolved by http
DEBUG  : primarydomain.com already resolved by http
DEBUG  : SSL challenge attempt: dns (*.addondomain.com)
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : Setting DNS TXT record _acme-challenge.addondomain.com with value tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : _acme-challenge.addondomain.com pdns dirty
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 1/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 2/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 3/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 4/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 5/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 6/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 7/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 8/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 9/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 10/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 11/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 12/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 13/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 14/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 15/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 16/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 17/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 18/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 19/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 20/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 21/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 22/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 23/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 24/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 25/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 26/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 27/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 28/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 29/30
DEBUG  : DNS record `_acme-challenge.addondomain.com' added asynchronously to ns1.myns.net - got `' want `tXTADy67sZROTwDf_7_kqcyYs_mrjq7B9kSLtGO5sc4' - wait 30/30
WARNING: Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): dns challenge failed: Challenge failed (response: {"type":"dns-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.addondomain.com - check that a DNS record exists for this domain","status":400},"url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/268773936546\/B9yl2g","token":"SeElImV5XO5zwJ7yaZmp4Az4_bq3jAQ46L1c8XpTwA8","validated":"2023-09-28T13:06:40Z"}).
ERROR  : Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): Failed to finalize certificate request: Certificate request failed (response: The order has not been validated)
         0. Error_Reporter::add_error("Failed to finalize certificate request: %s", ["Certificate request failed (response: The order has not been validated)"])
            [/usr/local/apnscp/lib/log_wrapper.php:62]
         1. error("Failed to finalize certificate request: %s", "Certificate request failed (response: The order has not been validated)")
            [/usr/local/apnscp/lib/Opcenter/Crypto/Letsencrypt/AcmeDispatcher.php:427]
         2. Opcenter\Crypto\Letsencrypt\AcmeDispatcher->solve(AcmePhp\Core\Protocol\CertificateOrder)
            [/usr/local/apnscp/lib/Opcenter/Crypto/Letsencrypt/AcmeDispatcher.php:175]
         3. Opcenter\Crypto\Letsencrypt\AcmeDispatcher->issue(["addondomain.com", "primarydomain.com", "*.addondomain.com", "*.primarydomain.com"])
            [/usr/local/apnscp/lib/Module/Support/Letsencrypt.php:260]
         4. Module\Support\Letsencrypt->requestReal(["addondomain.com", "primarydomain.com", "*.addondomain.com", "*.primarydomain.com"], "site6", true)
            [/usr/local/apnscp/lib/modules/letsencrypt.php:398]
         5. Letsencrypt_Module->request(["addondomain.com", "primarydomain.com", "*.addondomain.com", "*.primarydomain.com"], true, true)
            [/usr/local/apnscp/lib/modules/letsencrypt.php:516]
         6. Letsencrypt_Module->append([*.addondomain.com:0])
            [/usr/local/apnscp/lib/Module/Skeleton/Standard.php:145]
         7. Module\Skeleton\Standard->_invoke("append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/apnscpfunction.php:992]
         8. apnscpFunctionInterceptor->call("letsencrypt_append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/CLI/cmd.php:62]
         9. CLI\__call("letsencrypt_append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/CLI/cmd.php:581]
        10. CLI\main()
            [/usr/local/apnscp/bin/cmd:7]
ERROR  : Letsencrypt_Module::request(): Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): dns challenge failed: Challenge failed (response: {"type":"dns-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.addondomain.com - check that a DNS record exists for this domain","status":400},"url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/268773936546\/B9yl2g","token":"SeElImV5XO5zwJ7yaZmp4Az4_bq3jAQ46L1c8XpTwA8","validated":"2023-09-28T13:06:40Z"}).
         0. Error_Reporter::add_error("Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): dns challenge failed: Challenge failed (response: {"type":"dns-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.addondomain.com - check that a DNS record exists for this domain","status":400},"url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/268773936546\/B9yl2g","token":"SeElImV5XO5zwJ7yaZmp4Az4_bq3jAQ46L1c8XpTwA8","validated":"2023-09-28T13:06:40Z"...")
            [/usr/local/apnscp/lib/Module/Support/Letsencrypt.php:271]
         1. Module\Support\Letsencrypt->requestReal(["addondomain.com", "primarydomain.com", "*.addondomain.com", "*.primarydomain.com"], "site6", true)
            [/usr/local/apnscp/lib/modules/letsencrypt.php:398]
         2. Letsencrypt_Module->request(["addondomain.com", "primarydomain.com", "*.addondomain.com", "*.primarydomain.com"], true, true)
            [/usr/local/apnscp/lib/modules/letsencrypt.php:516]
         3. Letsencrypt_Module->append([*.addondomain.com:0])
            [/usr/local/apnscp/lib/Module/Skeleton/Standard.php:145]
         4. Module\Skeleton\Standard->_invoke("append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/apnscpfunction.php:992]
         5. apnscpFunctionInterceptor->call("letsencrypt_append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/CLI/cmd.php:62]
         6. CLI\__call("letsencrypt_append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/CLI/cmd.php:581]
         7. CLI\main()
            [/usr/local/apnscp/bin/cmd:7]
ERROR  : Letsencrypt_Module::request(): Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): Failed to finalize certificate request: Certificate request failed (response: The order has not been validated)
         0. Error_Reporter::add_error("Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): Failed to finalize certificate request: Certificate request failed (response: The order has not been validated)")
            [/usr/local/apnscp/lib/Module/Support/Letsencrypt.php:271]
         1. Module\Support\Letsencrypt->requestReal(["addondomain.com", "primarydomain.com", "*.addondomain.com", "*.primarydomain.com"], "site6", true)
            [/usr/local/apnscp/lib/modules/letsencrypt.php:398]
         2. Letsencrypt_Module->request(["addondomain.com", "primarydomain.com", "*.addondomain.com", "*.primarydomain.com"], true, true)
            [/usr/local/apnscp/lib/modules/letsencrypt.php:516]
         3. Letsencrypt_Module->append([*.addondomain.com:0])
            [/usr/local/apnscp/lib/Module/Skeleton/Standard.php:145]
         4. Module\Skeleton\Standard->_invoke("append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/apnscpfunction.php:992]
         5. apnscpFunctionInterceptor->call("letsencrypt_append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/CLI/cmd.php:62]
         6. CLI\__call("letsencrypt_append", ["*.addondomain.com"])
            [/usr/local/apnscp/lib/CLI/cmd.php:581]
         7. CLI\main()
            [/usr/local/apnscp/bin/cmd:7]
----------------------------------------
MESSAGE SUMMARY
Reporter level: ERROR
ERROR: Letsencrypt_Module::request(): Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): dns challenge failed: Challenge failed (response: {"type":"dns-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.addondomain.com - check that a DNS record exists for this domain","status":400},"url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/268773936546\/B9yl2g","token":"SeElImV5XO5zwJ7yaZmp4Az4_bq3jAQ46L1c8XpTwA8","validated":"2023-09-28T13:06:40Z"}).
ERROR: Letsencrypt_Module::request(): Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): Failed to finalize certificate request: Certificate request failed (response: The order has not been validated)
----------------------------------------

Unbound:

Query results for TXT addondomain.com

Response:
;; opcode: QUERY, status: NOERROR, id: 48867
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: do; udp: 512

;; QUESTION SECTION:
;addondomain.com.	IN	 TXT

;; ANSWER SECTION:
addondomain.com.	0	IN	TXT	"v=spf1 a mx include:relay.mailchannels.net ?all"

----- Unbound logs -----
Sep 28 13:13:16 unbound[1516568:0] notice: init module 0: validator
Sep 28 13:13:16 unbound[1516568:0] notice: init module 1: iterator
Sep 28 13:13:16 unbound[1516568:0] info: start of service (unbound 1.16.3).
Sep 28 13:13:17 unbound[1516568:0] info: 127.0.0.1 addondomain.com. TXT IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving addondomain.com. TXT IN
Sep 28 13:13:17 unbound[1516568:0] info: priming . IN NS
Sep 28 13:13:17 unbound[1516568:0] info: response for . NS IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 192.58.128.30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: priming successful for . NS IN
Sep 28 13:13:17 unbound[1516568:0] info: response for addondomain.com. TXT IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 2001:500:9f::42#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for addondomain.com. TXT IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <com.> 2001:501:b1f9::30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: resolving ns1.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving ns2.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving ns2.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving ns1.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: response for ns2.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 2001:500:a8::e#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for ns1.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 2001:500:a8::e#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for ns1.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 2001:dc3::35#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for ns2.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <net.> 2001:503:a83e::2:30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for ns1.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <net.> 192.12.94.30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for ns1.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <net.> 2001:500:d937::30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for ns2.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <myns.net.> 104.238.162.256#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: response for ns1.myns.net. A IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <myns.net.> 104.238.162.256#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: response for ns1.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <myns.net.> 45.77.110.256#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: response for ns2.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 2001:500:12::d0d#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: response for addondomain.com. TXT IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <addondomain.com.> 45.77.110.256#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: prime trust anchor
Sep 28 13:13:17 unbound[1516568:0] info: generate keytag query _ta-4f66. NULL IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving . DNSKEY IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving _ta-4f66. NULL IN
Sep 28 13:13:17 unbound[1516568:0] info: response for ns1.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <myns.net.> 104.238.162.256#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was nodata ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: response for _ta-4f66. NULL IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 198.41.0.4#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was NXDOMAIN ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: response for . DNSKEY IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <.> 2001:503:c27::2:30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: validate keys with anchor(DS): sec_status_secure
Sep 28 13:13:17 unbound[1516568:0] info: Successfully primed trust anchor . DNSKEY IN
Sep 28 13:13:17 unbound[1516568:0] info: validated DS com. DS IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving com. DNSKEY IN
Sep 28 13:13:17 unbound[1516568:0] info: response for ns2.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <net.> 192.54.112.30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was REFERRAL
Sep 28 13:13:17 unbound[1516568:0] info: resolving ns1.myns.net. AAAA IN
Sep 28 13:13:17 unbound[1516568:0] info: response for com. DNSKEY IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <com.> 2001:503:eea3::30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: validated DNSKEY com. DNSKEY IN
Sep 28 13:13:17 unbound[1516568:0] info: resolving addondomain.com. DS IN
Sep 28 13:13:17 unbound[1516568:0] info: response for addondomain.com. DS IN
Sep 28 13:13:17 unbound[1516568:0] info: reply from <com.> 192.35.51.30#53
Sep 28 13:13:17 unbound[1516568:0] info: query response was nodata ANSWER
Sep 28 13:13:17 unbound[1516568:0] info: NSEC3s for the referral proved no DS.
Sep 28 13:13:17 unbound[1516568:0] info: Verified that unsigned response is INSECURE```

Troy · September 28, 2023, 1:23pm

I was able to add wildcard for the primary domain, but not the addon domain.

msaladna · September 28, 2023, 3:04pm

You’d want to check TXT for _acme-challenge.addondomain.com, not addondomain.com.

Troy · October 4, 2023, 3:16pm

This keeps happening on multiple servers.
Everything that I can see checks out. TXT record is created and confirmed.
I’m not sure what to do, customers are getting frustrated and I don’t have a solution for them but to keep trying.

msaladna · October 4, 2023, 3:51pm

I understand this can be a very difficult time for you and your customers. I still need that unbound log against _acme-challenge.DOMAIN to have a better picture of what I am looking at. Nothing will be actionable until I have the information in front of me.

I know from what I run - BIND - renewals work fine unless Let’s Encrypt fails a CAA lookup using UDP. This may or may not be a contributing cause; it is impossible to know without the requisite diagnostics.

If this update on edge doesn’t fix it, verify the TXT record via unboundtest.com next time you spot it. This configuration is similar to what Let’s Encrypt uses.

env DEBUG=1 cpcmd -d DOMAIN letsencrypt:append ‘*.DOMAIN’

If this fails through the GUI, provide the GUI postback results. Again, to reiterate: I cannot proceed forward until I have more information.

Troy · April 22, 2024, 12:14am

Postback from GUI:
Action succeeded

reloading web server in 2 minutes, stay tuned!
reminder: only 5 duplicate certificates and 50 unique certificates may be issued per week per account

Unboundtest.com:
https://unboundtest.com/m/TXT/_acme-challenge.lithiumpanel.com/UN6CJQK3

From the GUI:

[root@p1 ~]# env DEBUG=1 cpcmd -d lithiumpanel.com letsencrypt:append '*.lithiumpanel.com'
DEBUG  : lithiumpanel.com already resolved by http
DEBUG  : SSL challenge attempt: dns (*.lithiumpanel.com)
DEBUG  : _acme-challenge.lithiumpanel.com pdns dirty
DEBUG  : Setting DNS TXT record _acme-challenge.lithiumpanel.com with value K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY
DEBUG  : _acme-challenge.lithiumpanel.com pdns dirty
DEBUG  : _acme-challenge.lithiumpanel.com pdns dirty
DEBUG  : _acme-challenge.lithiumpanel.com pdns dirty
DEBUG  : _acme-challenge.lithiumpanel.com pdns dirty
DEBUG  : _acme-challenge.lithiumpanel.com pdns dirty
DEBUG  : _acme-challenge.lithiumpanel.com pdns dirty
DEBUG  : _acme-challenge.lithiumpanel.com pdns dirty
DEBUG  : _acme-challenge.lithiumpanel.com pdns dirty
DEBUG  : _acme-challenge.lithiumpanel.com pdns dirty
DEBUG  : _acme-challenge.lithiumpanel.com pdns dirty
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 1/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 2/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 3/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 4/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 5/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 6/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 7/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 8/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 9/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 10/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 11/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 12/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 13/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 14/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 15/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 16/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 17/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 18/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 19/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 20/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 21/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 22/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 23/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 24/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 25/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 26/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 27/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 28/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 29/30
DEBUG  : DNS record `_acme-challenge.lithiumpanel.com' added asynchronously to ns1.lithiumdns.net - got `' want `K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY' - wait 30/30
WARNING: Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): dns challenge failed: Challenge failed (response: {"type":"dns-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:dns","detail":"During secondary validation: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.lithiumpanel.com - check that a DNS record exists for this domain","status":400},"url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/341420474147\/zeZiBA","token":"99TO-nCPSAj8F9YPIuuFas8ztOkerixTLIZDdwEXHRo","validationRecord":[{"hostname":"lithiumpanel.com","resolverAddrs":["10.1.12.84:21929"]}],"validated":"2024-04-22T00:13:06Z"}).
DEBUG  : Retrying request in non-strict mode. Pruned *.lithiumpanel.com
WARNING: Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): Following hostnames pruned to satisfy request: *.lithiumpanel.com
INFO   : reloading web server in 2 minutes, stay tuned!
INFO   : reminder: only 5 duplicate certificates and 50 unique certificates may be issued per week per account
----------------------------------------
MESSAGE SUMMARY
Reporter level: SUCCESS
INFO: reloading web server in 2 minutes, stay tuned!
INFO: reminder: only 5 duplicate certificates and 50 unique certificates may be issued per week per account
----------------------------------------
1
[root@p1 ~]#

Troy · April 22, 2024, 12:15am

It’s happened quite a bit since the last reply but typically doing the CLI route would work. This time it actually didn’t, the only SSL domain is lithiumpanel.com, no subdomains or wildcard on the cert. So this issue persists and I don’t know why.

msaladna · April 22, 2024, 12:28am

What does dig +norec txt @ns1.lithiumdns.net _acme-challenge.lithiumpanel.com report from the server making the request?

Troy · April 22, 2024, 12:30am

[root@p1 ~]# dig +norec txt @ns1.lithiumdns.net _acme-challenge.lithiumpanel.com

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.15 <<>> +norec txt @ns1.lithiumdns.net _acme-challenge.lithiumpanel.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36096
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme-challenge.lithiumpanel.com. IN   TXT

;; ANSWER SECTION:
_acme-challenge.lithiumpanel.com. 120 IN TXT    "K7QCxscRNBkBWNLO0oXaU-XSsAbHej-Qf1u1HnfIJTY"

;; Query time: 18 msec
;; SERVER: 104.238.162.214#53(104.238.162.214)
;; WHEN: Sun Apr 21 20:29:49 EDT 2024
;; MSG SIZE  rcvd: 117

msaladna · April 22, 2024, 12:49am

I’m getting a positive response internally. Try renewing it again. If it fails, immediately run dig +norec txt @ns1.lithiumdns.net _acme-challenge.lithiumpanel.com to determine if PowerDNS is caching the response.

Edit: and to be clear, the unbound test was run immediately after the request posted failed? It’s reporting something entirely different from K7QCxscRNBkBWNLO0oXaU-....

Troy · April 22, 2024, 1:08am

Yes the unbound test was run right away.

GUI Request failed, wiped all subdomain certs and then left just the main domain.
Ran the command again and received this response. WTF!?

[root@p1 ~]# dig +norec txt @ns1.lithiumdns.net _acme-challenge.lithiumpanel.com

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.15 <<>> +norec txt @ns1.lithiumdns.net _acme-challenge.lithiumpanel.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 16858
;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme-challenge.lithiumpanel.com. IN   TXT

;; AUTHORITY SECTION:
lithiumpanel.com.       600     IN      SOA     ns1.lithiumdns.net. hostmaster.lithiumpanel.com. 2024042206 3600 1800 604800 600

;; Query time: 18 msec
;; SERVER: 104.238.162.214#53(104.238.162.214)
;; WHEN: Sun Apr 21 21:07:29 EDT 2024
;; MSG SIZE  rcvd: 126

Troy · April 22, 2024, 1:09am

Ran the command again and it returned the TXT record with a different value than before which is to be expected I’m sure.