Let's Encrypt renewal failure logs

Discussion thread for feature request with the same title so we can debug or rant about stuff here instead.

Should be resolved as of the re-release of 3.2.45 from this morning.

Just to mention, in my last SSL incident I had a node with net.hostname cert expired as well so there’s something that went really wrong for some reason and we can’t know what.

Renewing via env DEBUG=1 cpcmd letsencrypt:request "$(cpcmd scope:get net.hostname)" did work and I went ahead renewing manually the other broken sites too.

Apparently no reason for hostname to fail as DNS hasn’t changed in years, domain wasn’t expiring and no other overlapping definitions exist.

Looking forward to the 3.2.45 then :crossed_fingers:

Adding on to this, am running 3.2.45 and didn’t have at least one site renew. Ran the debug command env DEBUG=1 cpcmd letsencrypt:renew-expiring and got the following output:

[root@s1 ~]# env DEBUG=1 cpcmd letsencrypt:renew-expiring
DEBUG  : Inspecting cert site1 for renewal
DEBUG  : Certificate site1 not within expiry threshold. Expires in 84 days, expiry range [0, 10].
DEBUG  : Inspecting cert MAIN for renewal
DEBUG  : SSL challenge attempt: http (s1.domain.com)
DEBUG  : query response from s1.domain.com (1.0.0.1): X.X.X.X (censored)
DEBUG  : HTTP: setting `vRCtR3isscS6prBUnZpzRlqjMbF0lclK0gYLa27V42k.eVlB-pNEkoJWfs_UHsTzipSw0lMSBPd6qI-bykZMCCs' in `/tmp/acme/.well-known/acme-challenge/vRCtR3isscS6prBUnZpzRlqjMbF0lclK0gYLa27V42k'
DEBUG  : SUCCESS! SSL challenge response: s1.domain.com (http) - VALID
INFO   : System certificate updated
DEBUG  : Running hooks for `' (user: `theadmin')
DEBUG  : missing `reload' hook on `siteinfo' module
DEBUG  : missing `reload' hook on `billing' module
DEBUG  : missing `reload' hook on `dns' module
DEBUG  : missing `reload' hook on `ipinfo' module
DEBUG  : missing `reload' hook on `ipinfo6' module
DEBUG  : missing `reload' hook on `mysql' module
DEBUG  : missing `reload' hook on `cgroup' module
DEBUG  : 0.06796: Web_Module -> _reload
DEBUG  : missing `reload' hook on `bandwidth' module
DEBUG  : missing `reload' hook on `ssl' module
DEBUG  : missing `reload' hook on `tomcat' module
DEBUG  : missing `reload' hook on `users' module
DEBUG  : missing `reload' hook on `aliases' module
DEBUG  : missing `reload' hook on `auth' module
DEBUG  : 0.02315: Ftp_Module -> _reload
DEBUG  : missing `reload' hook on `pgsql' module
DEBUG  : missing `reload' hook on `ssh' module
DEBUG  : 0.10314: Email_Module -> _reload
DEBUG  : 0.00948: Crontab_Module -> _reload
DEBUG  : missing `reload' hook on `mlist' module
DEBUG  : missing `reload' hook on `spamfilter' module
DEBUG  : missing `reload' hook on `dav' module
DEBUG  : 0.00006: Letsencrypt_Module -> _reload
DEBUG  : 0.00002: Php_Module -> _reload
DEBUG  : missing `reload' hook on `sql' module
DEBUG  : missing `reload' hook on `webapp' module
----------------------------------------
MESSAGE SUMMARY
Reporter level: SUCCESS
INFO: System certificate updated
DEBUG: Running hooks for `' (user: `theadmin')
DEBUG: missing `reload' hook on `siteinfo' module
DEBUG: missing `reload' hook on `billing' module
DEBUG: missing `reload' hook on `dns' module
DEBUG: missing `reload' hook on `ipinfo' module
DEBUG: missing `reload' hook on `ipinfo6' module
DEBUG: missing `reload' hook on `mysql' module
DEBUG: missing `reload' hook on `cgroup' module
DEBUG: 0.06796: Web_Module -> _reload
DEBUG: missing `reload' hook on `bandwidth' module
DEBUG: missing `reload' hook on `ssl' module
DEBUG: missing `reload' hook on `tomcat' module
DEBUG: missing `reload' hook on `users' module
DEBUG: missing `reload' hook on `aliases' module
DEBUG: missing `reload' hook on `auth' module
DEBUG: 0.02315: Ftp_Module -> _reload
DEBUG: missing `reload' hook on `pgsql' module
DEBUG: missing `reload' hook on `ssh' module
DEBUG: 0.10314: Email_Module -> _reload
DEBUG: 0.00948: Crontab_Module -> _reload
DEBUG: missing `reload' hook on `mlist' module
DEBUG: missing `reload' hook on `spamfilter' module
DEBUG: missing `reload' hook on `dav' module
DEBUG: 0.00006: Letsencrypt_Module -> _reload
DEBUG: 0.00002: Php_Module -> _reload
DEBUG: missing `reload' hook on `sql' module
DEBUG: missing `reload' hook on `webapp' module
----------------------------------------

I also ran (per the discussion this post references): cpcmd scope:set cp.config letsencrypt lookbehind_days -10 && systemctl restart apiscp and then reran the above command, and got the following output:

[root@s1 ~]# cpcmd scope:set cp.config letsencrypt lookbehind_days -10 && systemctl restart apiscp
INFO   : ApisCP will restart in 1 minute
----------------------------------------
MESSAGE SUMMARY
Reporter level: SUCCESS
INFO: ApisCP will restart in 1 minute
----------------------------------------
1
[root@s1 ~]# 
[root@s1 ~]# env DEBUG=1 cpcmd letsencrypt:renew-expiring
DEBUG  : Inspecting cert site1 for renewal
DEBUG  : Certificate site1 not within expiry threshold. Expires in 84 days, expiry range [-10, 10].
DEBUG  : Inspecting cert MAIN for renewal
DEBUG  : Certificate MAIN not within expiry threshold. Expires in 89 days, expiry range [-10, 10].
----------------------------------------
MESSAGE SUMMARY
Reporter level: SUCCESS
DEBUG: Inspecting cert site1 for renewal
DEBUG: Certificate site1 not within expiry threshold. Expires in 84 days, expiry range [-10, 10].
DEBUG: Inspecting cert MAIN for renewal
DEBUG: Certificate MAIN not within expiry threshold. Expires in 89 days, expiry range [-10, 10].
----------------------------------------

The SSL cert still hadn’t been renewed.
Upon running env DEBUG=1 cpcmd -d site7 letsencrypt:renew the process DID complete with no errors and that site’s SSL has been renewed, but it seems the issue of auto-renewal still exists.

Run bin/scripts/reissueAllCertificates.php --dry-run. If the certificate isn’t located within storage/certificates, then it won’t be picked up by letsencrypt:renew-expiring. reissueAllCertificates.php performs an import if it’s a Let’s Encrypt certificate now.

Should there have been any output on that command?

Only when run with env DEBUG=1. It’ll perform an import into storage/certificates// for each certificate missing.