Let's Encrypt renewal is impossible for accounts whose plan disables Apache ("Certificate is not Let's Encrypt")

Environment

ApisCP:

ApisCP v3.2.0
revision: 75f8143eb3311984028ce9f7406e42694f11e75c (2026/09/25)
PHP 8.3.24

Affects any account on a plan with apache enabled=0.

Why an email-only account has a certificate at all. An account with Apache
disabled still needs a TLS certificate. Mail is offered over IMAPS (993), POP3S
(995) and SMTPS (465), terminated by HAProxy, and mail clients reject an expired
or untrusted certificate exactly as a browser would. Disabling Apache removes the
web server, not the need for the certificate — so the certificate is legitimately
issued and installed against an account that has no web vhost.

That is precisely the combination that breaks renew() below.

The ssl service value is not the deciding factor, which is worth stating
because it is the first thing to reach for. ssl_install()
(lib/modules/ssl.php:360, ssl::install()) enables the service on the account
as part of installing a certificate:

// pre-flight checks done, let's install
if (!$overwrite || !$this->enabled()) {
    $cmd = new Util_Account_Editor($this->getAuthContext()->getAccount(), $this->getAuthContext());
    $cmd->setConfig(SiteConfiguration::getModuleRemap('openssl'), 'enabled', 1);
    // ensure HTTP config is rebuild
    $cmd->edit();
}

So an email account carries a live, in-use certificate regardless of what its
plan sets. Setting ssl enabled=1 in the plan — which we now do, and which is
correct — does not avoid the bug: the lookup that fails depends on the
account having an Apache vhost, and apache enabled=0 prevents that by design.
Verified on the affected account after enabling ssl in the plan:

$ cpcmd -d email-only.com.au -i json ssl_get_certificates
{  }                                    # apache=0, ssl=1, certificate in active use

$ cpcmd -d web-site.com.au -i json ssl_get_certificates
- key: server.key                       # apache=1
  crt: server.crt
  chain: bundle.crt
  host: 221.121.144.23

Summary

letsencrypt:renew cannot renew the certificate on an account whose plan
disables Apache. It reports:

WARNING: Letsencrypt_Module::renew(): Certificate is not Let's Encrypt

The certificate is Let’s Encrypt. The message describes a failed lookup, not
the certificate’s issuer, so it points away from the real cause. The account
never renews, and the failure is silent — it only becomes visible roughly 80
days later when mail clients start rejecting an expired certificate.

Steps to reproduce

  1. Create an account that mirrors the email plans above — Apache disabled, with a
    certificate for mail:

    AddDomain -c siteinfo,domain=example.com \
              -c siteinfo,admin_user=example \
              -c auth,passwd=1 \
              -c apache,enabled=0 \
              -c ssl,enabled=1
    
  2. Issue a certificate, working around the apex/IP check if the domain’s A
    record points elsewhere:

    cpcmd -d example.com letsencrypt_request '["*.example.com"]' false
    

    This succeeds:

    INFO   : successfully renewed certificate for 90 days
    Reporter level: SUCCESS
    
  3. Now attempt a renewal — the path housekeeping takes automatically:

    cpcmd -d example.com letsencrypt_renew
    

Actual result

WARNING: Letsencrypt_Module::renew(): Certificate is not Let's Encrypt
----------------------------------------
MESSAGE SUMMARY
Reporter level: WARNING
WARNING: Letsencrypt_Module::renew(): Certificate is not Let's Encrypt
----------------------------------------

Nothing is renewed. No error is raised, and nothing is written to the log beyond
this warning.

Root cause

renew() resolves the hostnames to re-request via
getNonOrphanedDomainsFromCertificate(), which calls getSanFromCertificate()
with no argument (lib/modules/letsencrypt.php:534):

private function getNonOrphanedDomainsFromCertificate(): ?array
{
    if (null === ($cns = $this->getSanFromCertificate())) {
        // not Let's Encrypt
        return null;
    }

getSanFromCertificate() (lib/Module/Support/Letsencrypt.php:375, pulled into
the module via use Module\Support\Letsencrypt;) only consults the ACME storage
directory when it is given an explicit $cert argument. With no argument it
falls through to ssl_get_certificate():

protected function getSanFromCertificate(?string $cert = null): ?array
{
    if (!$cert && $this->permission_level & PRIVILEGE_ADMIN) {
        $cert = LEService::SYSCERT_NAME;
    }
    if (!$cert) {
        if (!$cert = $this->ssl_get_certificate()) {   // <-- reads the Apache vhost
            return null;
        }
    } else if (is_dir($path = LEService::acmeSiteStorageDirectory($cert))) {
        $components = LEService::getCertificateComponentData($cert);
        $cert = $components['crt'] . "\n" . $components['chain'];
    } else {
        return null;
    }

ssl_get_certificate() derives the certificate from the account’s Apache
vhost
— ssl_get_certificates() parses /etc/httpd/conf/virtual/<site>, which
is surfaced into conf/virtual-httpd-built. An account with
apache enabled=0 has no vhost, so:

cpcmd -d example.com -i json ssl_get_certificates

returns { } even though the certificate is installed and in active use for
mail. getSanFromCertificate() therefore returns null, and renew() takes
the // not Let's Encrypt branch.

The class already calls the correct form elsewhere — filterMissingHostnames()
(lib/Module/Support/Letsencrypt.php:440) passes the site explicitly at line 442:

$new = array_flip((array)$this->getSanFromCertificate($this->site));

which resolves through the is_dir(acmeSiteStorageDirectory(...)) branch and
works regardless of whether Apache is enabled. getNonOrphanedDomainsFromCertificate()
omitting the argument looks like an oversight rather than a deliberate choice.

Expected result

renew() should resolve the certificate’s SANs for site-level contexts without
depending on a web vhost existing, and renew successfully — or, if a vhost
really is required, fail loudly with a message that names the actual problem
instead of reporting a certificate issuer mismatch.

Proposed fix

Pass the site through for site contexts, while preserving the admin behaviour of
resolving to the MAIN system certificate:

 private function getNonOrphanedDomainsFromCertificate(): ?array
 {
+    // Resolve from the ACME storage directory for site contexts. Without an
+    // explicit argument getSanFromCertificate() reads the certificate out of
+    // the site's Apache vhost, which does not exist for accounts whose plan
+    // disables apache (email-only, DNS-only) - renew() then reports
+    // "Certificate is not Let's Encrypt" and never renews.
+    $cert = $this->permission_level & PRIVILEGE_ADMIN ? null : $this->site;
-    if (null === ($cns = $this->getSanFromCertificate())) {
+    if (null === ($cns = $this->getSanFromCertificate($cert))) {
         // not Let's Encrypt
         return null;
     }

The ternary is deliberate: passing $this->site unconditionally would break
admin contexts, which have no site and must still resolve to MAIN.

Verification

Applied to lib/modules/letsencrypt.php on the 3.2.0 install described above,
php -l clean, systemctl restart apnscp.service, then:

# cpcmd -d email-only.com.au letsencrypt_renew
DEBUG  : *.email-only.com.au already resolved by dns
INFO   : reloading web server in 2 minutes, stay tuned!
INFO   : reminder: only 5 duplicate certificates and 50 unique certificates may be issued per week per account
INFO   : successfully renewed certificate for 90 days
----------------------------------------
MESSAGE SUMMARY
Reporter level: SUCCESS

Before the patch the same command produced only the Certificate is not Let's Encrypt warning and issued nothing. The account in question is apache enabled=0, ssl enabled=1, with no Apache vhost present.

Impact

Any account whose plan disables Apache cannot renew its certificate. That
includes email-only and DNS-only plans, which are a common way to sell mail
without web hosting — and which still require a valid certificate, because the
mail ports present one to every connecting client.

The failure is silent and delayed. Renewal returns without an error, so nothing
surfaces until roughly 80 days later when the certificate expires and mail
clients begin rejecting it. Because the warning reads as an issuer
classification rather than a lookup failure, the natural troubleshooting
direction is to inspect the certificate, which is valid and correct; the account
configuration is what actually matters.

Related observations

These are separate from the bug above but surfaced during the same
investigation, and both also affect email-only accounts.

1. verifyip passed as a command-line argument does not persist into
renewal.
letsencrypt_request '["*.example.com"]' false sets verifyip for
that single call only. renew() re-reads the preference, falling back to
LETSENCRYPT_VERIFY_IP (true), so any domain whose apex resolves elsewhere
(a website on Squarespace, registrar parking, a separate mail host) is rejected
during renewal as not resolving to this server. The durable form is:

cpcmd -d example.com common_set_preference letsencrypt.verifyip false

Worth noting because verifyip=false on the issuance command looks like a
persisted setting and is not — a caller can reasonably believe the account is
configured for renewal when it is not.

2. A successful renewal does not redeploy to HAProxy. Mail TLS on
993/465/995 is terminated by HAProxy from /etc/haproxy/ssl.d/<site>.pem, which
is a different file from the web certificate. On a successful renewal the site’s
httpd path was updated but the proxy pem was not:

/home/virtual/site31/fst/etc/httpd/conf/ssl.crt/server.crt   serial=06A6C23D...
/etc/haproxy/ssl.d/site31.pem                                serial=06642851...   # stale

Mail kept serving the previous certificate after a renewal that reported
Reporter level: SUCCESS. In this case the site was email-only, so the web path
was not in use and the mismatch was only detectable by comparing serials. If the
email::merge_ssl() hook that is meant to sync this is expected to cover
renewals as well as initial issuance, it did not fire here.

The AI-proposed fix is bunk.

Relocating per-site SSL to /etc/ssl within vfs is ideal. It preserves existing code pathways while decoupling ssl module from apache. We can’t do /etc/pki within vfs as that’s linked to a shared system path, /.socket.

I’ll look into this further next week. Enable apache for affected sites as a temporary workaround until then.

1. verifyip passed as a command-line argument does not persist into
renewal.
letsencrypt_request '["*.example.com"]' false sets verifyip for that single call only.

This is intended behavior. If the parameter is omitted then the account preference is used. Generally, API methods should minimize side-effects, including implicitly rewriting preferences unless that is its expressed function.

2. A successful renewal does not redeploy to HAProxy.

lgtm. Rocky 10. Cannot reproduce.

# cpcmd -d site2 letsencrypt:renew
# ... verifications ...
----------------------------------------
MESSAGE SUMMARY
Reporter level: SUCCESS
INFO: reloading web server in 2 minutes, stay tuned!
INFO: reminder: only 5 duplicate certificates and 50 unique certificates may be issued per week per account
INFO: successfully renewed certificate for 90 days
----------------------------------------


# at -l
189     Wed Oct  7 14:55:00 2026 a root
190     Wed Oct  7 14:55:00 2026 a root
191     Wed Oct  7 14:55:00 2026 a root
192     Wed Oct  7 14:55:00 2026 a root

# at -c 189 19{0,1,2} | grep systemctl
/etc/systemd/user/httpd.init buildconfig && /usr/bin/systemctl  reload httpd
/usr/bin/systemctl  reload haproxy
/usr/bin/systemctl  restart dovecot
/usr/bin/systemctl  restart postfix

# openssl x509 -noout -serial -in /home/virtual/site2/fst/etc/httpd/conf/ssl.crt/server.crt
serial=05D888197B9E7E86B01F63B642C27857640C
# openssl x509 -noout -serial -in /etc/haproxy/ssl.d/site2.pem
serial=05D888197B9E7E86B01F63B642C27857640C

That’s on me. I assumed it was just a quick fix, rather than something that might need a bit more architectural work. I might throw a couple of agents on it today and see if I can come up with something.

Proposed Implementation

A solution has been developed that implements per-site certificates in /etc/ssl… The code and patches are too big to post here, but I’m happy to provide them.

On accounts with apache=0 + ssl=1 (email-only/DNS-only), certificate issuance/renewal, ssl_get_certificates(), ssl_cert_exists(), and the HAProxy mail certificate all break, and a wildcard cert (required for email-only hosting) can never be issued when the apex points elsewhere. All addressed and verified end-to-end on a live AlmaLinux 10 install.

Root causes

  1. ssl coupled to Apache. Ssl_Module::get_certificates() discovered certs by parsing the Apache vhost — absent for apache=0 — so cert_exists, letsencrypt::renew(), email::merge_ssl(), checkPhantomCertificate() and letsencrypt::_reload() all failed. _reload() (fired on every ssl_install) then saw no cert, concluded it wasn’t Let’s Encrypt, and rmdir’d the ACME storage — issue once, never renew.
  2. Wildcards required an apex A record. _verifyIP() ran against the apex even for a *.example.com DNS-01 challenge, which doesn’t need it → wildcard dropped/aborted for split-hosting.
  3. is_ca() on a lone leaf fails for newer intermediates (YE2/YE absent from the OS CA store); the deeper chain walk needs the intermediate.
  4. DNS-01 wait too low and per-nameserver (30s default, initial local wait ignored) — brittle for external replication.
  5. Config builder substitutes the system cert. httpd.init’s collapse_config scans virtual*/<site> and substitutes /etc/httpd/conf/server.pem for any SSL* line whose file is missing; it does not re-render per-site vhosts, so moving certs without updating stored vhost paths made Apache serve the system certificate.

Fixes

Decouple ssl from Apache (flat /etc/ssl)

  • Ssl_Module::SSL_PATH = '/etc/ssl' (CRT_PATH/KEY_PATH/CSR_PATH kept as deprecated aliases); certPath()/keyPath()/storagePath() with legacy fallback.
  • get_certificates() reads /etc/ssl/server.{crt,key} directly (no vhost parse).
  • Ssl::absolutePath()/unify() updated; install() refuses a symlinked /etc/ssl; DEPENDENCY_MAP drops apache; Provisioning\Ssl::TEMPLATE_DIRECTORIES emptied; virtualhost-ssl.blade.php uses the helpers.

Renewal durability

With the vhost-independent reader, _reload() and checkPhantomCertificate() work and no longer purge ACME storage.

Wildcards

_verifyIP() skips the apex check for wildcard CNs (+ bounded backoff); bootstrap() honours letsencrypt.verifyip.

CA detection

is_ca() fed leaf + chain in the relocation and reissue scripts.

DNS-01 wait

Single aggregate deadline across nameservers; default 120s, exposed as scope ssl.dns-validation-wait.

Migration

bin/scripts/relocateCertificates.php + migration YAML: moves artifacts flat into /etc/ssl, rewrites the <site>.ssl/custom chain directive and the stored per-site vhost cert paths (fixes #5), re-imports missing ACME storage, then Apache::activate(). Idempotent, --dry-run, --site=<id|domain>.

Misc

  • letsencrypt::solve() no longer discards its solver arg.
  • systemNeedsIssuance() uses array_merge (previously dropped additional_certs[0]).
  • email::merge_ssl() legacy-aware fallback.
  • Admin get_certificate(null) / key_exists path bugs.

Verification

  • ssl_get_certificates returns the cert; ssl_cert_exists/letsencrypt_exists = 1; ACME storage present.
  • Forced production renewal on the email-only wildcard account: DNS-01 issued, ACME storage survived _reload, HAProxy PEM refreshed.
  • Apache-enabled sites serve their own LE certs (verified via SNI); httpd -t OK; idempotent re-run reports 0 account(s).

Note on 8cce495b0

Your “include missing full chain during renewal inspection” commit addresses the same intermediate-absent-from-CA-store class; this extends it to the re-import/reissue recovery paths.

PR

Happy to open a PR against apisnetworks/apnscp with the SSL/LE commits:

Commit Subject
93357e5b7 FIX: SSL certificate storage independent of Apache, support apache-disabled plans (Ssl)
37bf3a053 FIX: detect Let’s Encrypt authority from leaf+chain during cert re-import (ssl)
adc853d67 FIX: drop backend-only reload hook from relocation script (ssl)
67be18778 FIX: rewrite per-site vhost cert paths during relocation (ssl)
036b9e0c2 FIX: harden cert relocation + reissue CA detection (ssl)

As a personal preference, I would like to ensure code passes through human stewardship. AI is fine for troubleshooting, guidance, and even peer review. Final code production needs to be vetted by human as a professional courtesy.

After all, if these servers blow up and rm -rf --no-preserve-root / gets injected through a nonsensical hallucination, I’m getting sued under myriad tort laws from US customers.

It’s not good for the end-user and certainly not good for my career… plus I don’t think my PLI policy covers it.

These patches can be submitted by PR. I’ll look at their suitability but unlikely to implement anything fully.