Wildcard SSL dropping on lots of domains, re-adding it fails.

msaladna · April 22, 2024, 1:33am

In /etc/pdns/pdns.conf, set query-cache-ttl=5. Restart pdns service on ns1, then give it another attempt.

Troy · April 22, 2024, 1:42am

I did that, from the GUI I get this, so I’ll try tomorrow.

 Action failed

Letsencrypt_Module::request(): Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): dns challenge failed: Challenge failed (response: {"type":"dns-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.lithiumpanel.com - check that a DNS record exists for this domain","status":400},"url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/341448557957\/VboTMQ","token":"EooFwgwliwK6_5KhUrok-iBu590w4cJVDfKOkdAf6AY","validated":"2024-04-22T01:41:23Z"}).
Letsencrypt_Module::request(): Opcenter\Crypto\Letsencrypt\AcmeDispatcher::issue(): Failed acquiring challenges for one or more domains: [rateLimited] This client reached the rate limit of the server: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: lithiumpanel.com, retry after 2024-04-23T09:43:19Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/ (on request "POST https://acme-v02.api.letsencrypt.org/acme/new-order")```

Troy · May 7, 2024, 5:32pm

It ended up working for that account but this same issue keeps cropping up for other accounts on various servers. What else can we do to try and resolve this once and for all?

Troy · August 1, 2024, 12:49pm

Brand new customer / account / domain and wildcard will not issue.
DNS has been fully setup for more than 48 hours and it still just won’t do wildcard.

msaladna · August 1, 2024, 12:57pm

And what does env DEBUG=1 report?

Troy · August 1, 2024, 1:19pm

It’s always the same, it cannot detect the txt record which is always set in advance. It’s nothing new, we’ve covered debug and unbound before. Just reporting this happens all the time and it’s so frustrating I’m about to edit the template and remove * from the drop down to be done with it.

msaladna · August 1, 2024, 1:24pm

Please provide the latest debug logs, and we’ll go from there. I prefer to collate related issues, which may create more work for you but allows me to be thorough.

Troy · August 1, 2024, 2:11pm

An error occurred: Body is limited to 32000 characters; you entered 214230.

Apparently can’t post large blocks of text.

Troy · August 1, 2024, 2:12pm

~]# env DEBUG=1 cpcmd -d domain.com letsencrypt:append '*.domain.com'
DEBUG  : domain.com already resolved by http
DEBUG  : SSL challenge attempt: dns (*.domain.com)
DEBUG  : _acme-challenge.domain.com pdns dirty
DEBUG  : _acme-challenge.domain.com pdns dirty
DEBUG  : _acme-challenge.domain.com pdns dirty
DEBUG  : _acme-challenge.domain.com pdns dirty
DEBUG  : Setting DNS TXT record _acme-challenge.domain.com with value BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw
DEBUG  : _acme-challenge.domain.com pdns dirty
DEBUG  : _acme-challenge.domain.com pdns dirty
DEBUG  : _acme-challenge.domain.com pdns dirty
DEBUG  : _acme-challenge.domain.com pdns dirty
DEBUG  : _acme-challenge.domain.com pdns dirty
DEBUG  : _acme-challenge.domain.com pdns dirty
DEBUG  : _acme-challenge.domain.com pdns dirty
DEBUG  : _acme-challenge.domain.com pdns dirty
DEBUG  : _acme-challenge.domain.com pdns dirty
DEBUG  : _acme-challenge.domain.com pdns dirty
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 1/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 2/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 3/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 4/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 5/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 6/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 7/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 8/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 9/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 10/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 11/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 12/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 13/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 14/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 15/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 16/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 17/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 18/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 19/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 20/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 21/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 22/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 23/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 24/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 25/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 26/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 27/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 28/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 29/30
DEBUG  : DNS record `_acme-challenge.domain.com' added asynchronously to ns2.lithiumdns.net - got `' want `BqJD-MzFCZj0BFpgbmfiN7Wdi098JPhb8dE3GZaVEiw' - wait 30/30
WARNING: Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): dns challenge failed: Challenge failed (response: {"type":"dns-01","url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/384704432537\/1VT-qw","status":"invalid","validated":"2024-08-01T14:08:16Z","error":{"type":"urn:ietf:params:acme:error:dns","detail":"During secondary validation: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.domain.com - check that a DNS record exists for this domain","status":400},"token":"QE4mvY9oNNw_fNLXQXbhtA-UXOhK4BSONN-kxpLdFSA","validationRecord":[{"hostname":"domain.com"}]}).
DEBUG  : Retrying request in non-strict mode. Pruned *.domain.com
WARNING: Opcenter\Crypto\Letsencrypt\AcmeDispatcher::solve(): Following hostnames pruned to satisfy request: *.domain.com
INFO   : reloading web server in 2 minutes, stay tuned!
INFO   : reminder: only 5 duplicate certificates and 50 unique certificates may be issued per week per account
----------------------------------------
MESSAGE SUMMARY
Reporter level: SUCCESS
INFO: reloading web server in 2 minutes, stay tuned!
INFO: reminder: only 5 duplicate certificates and 50 unique certificates may be issued per week per account
----------------------------------------
1

UNBOUND:

msaladna · August 1, 2024, 3:30pm

In the first case ns1 was reporting NXDOMAIN, in this instance ns2 is reporting NXDOMAIN.

Which version of PowerDNS for both servers? pdns_server --version

I’d also verify the TXT record is replicating, dig @ns2.lithiumhosting.com +norec _acme-challenge.domain.com. unbound confirms ns1 reports a record but ns2 is unknown.

Troy · August 1, 2024, 3:50pm

There’s no replication between servers, it’s a galera backend.

Troy · August 2, 2024, 12:24am

It’s Ubuntu 20.04 LTS
PowerDNS Authoritative Server 4.8.4 on all 3 servers.
One is the API server, the other two are NS1 and NS2.

All 3 servers using MySQL with a Galera replication (not a PowerDNS replication).

| wsrep_evs_repl_latency        | 0/0/0/0/0                                                  |
| wsrep_evs_state               | OPERATIONAL     
| wsrep_local_state_comment     | Synced       
| wsrep_protocol_version        | 9                                                          |
| wsrep_provider_name           | Galera                                                     |
| wsrep_provider_vendor         | Codership Oy <info@codership.com>                          |
| wsrep_provider_version        | 25.3.37(rd0a7bd74)                                         |
| wsrep_ready                   | ON

msaladna · August 2, 2024, 2:30pm

Unsupported distro and backend approach, which hinders my ability to adequately troubleshoot this. Last thing I’d recommend is reducing negquery-cache-ttl to 20 seconds as that would put all caches at 20 seconds or lower.

Beyond this, if I were debugging this, I’d enable query logging, Galera logging, and PowerDNS logging to look at timestamp offsets.

Bear in mind Galera is virtually synchronous; there is no guarantee the batch has propagated to members of the cluster. Enabling wsrep_sync_wait may rectify your situation but it’s purely a shot in the dark.