That’s possible, the server with the cp proxy was the one with the other issue. However, it should have renewed the certificate by yesterday. When I visit the site with the webbrowser, it shows an accurate certificate.
[root@s1 ~]# AddDomain -c siteinfo,domain=daskjldfsdfs.com -c siteinfo,admin_user=sdfkldfs -c dns,enabled=0
teDomain daskjldfsdfs.comINFO : no billing identifier specified for site, generated `apnscp-EWBXFYQO'
INFO : no mysql database prefix specified - suggesting `dask_'
INFO : no bandwidth rollover date specified, using day of month, `6'
INFO : No password specified - generating random password: `TuQrKccI2kiZLOsIDmC1ROMoMMSrSQ9P'
INFO : allocated port range [46070,46079] for account
WARNING: Email_Module::add_virtual_transport(): DNS is not configured for `daskjldfsdfs.com' - unable to provision DNS automatically
[root@s1 ~]# env DEBUG=1 DeleteDomain daskjldfsdfs.com
DEBUG : Running hooks for `daskjldfsdfs.com' (user: `sdfkldfs')
DEBUG : 0.00053: Webapp_Module -> _delete
DEBUG : 0.00002: Sql_Module -> _delete
DEBUG : 0.00114: Php_Module -> _delete
DEBUG : 0.00030: Letsencrypt_Module -> _delete
DEBUG : 0.00002: Dav_Module -> _delete
DEBUG : 0.00002: Spamfilter_Module -> _delete
DEBUG : 0.00335: Majordomo_Module -> _delete
DEBUG : Service config `crontab' disabled for site - disabling calling hook `_delete' on `Crontab_Module'
DEBUG : 0.00003: Email_Module -> _delete
DEBUG : 0.00001: Ssh_Module -> _delete
DEBUG : Service config `pgsql' disabled for site - disabling calling hook `_delete' on `Pgsql_Module'
DEBUG : 0.00001: Ftp_Module -> _delete
DEBUG : 2.12954: Auth_Module -> _delete
DEBUG : 0.00003: Aliases_Module -> _delete
DEBUG : 0.00099: User_Module -> _delete
DEBUG : Service config `tomcat' disabled for site - disabling calling hook `_delete' on `Tomcat_Module'
DEBUG : 0.00002: Ssl_Module -> _delete
DEBUG : 0.00056: Bandwidth_Module -> _delete
DEBUG : 0.00002: Web_Module -> _delete
DEBUG : 0.11761: Mysql_Module -> _delete
DEBUG : 0.00002: Cgroup_Module -> _delete
DEBUG : Service config `ipinfo6' disabled for site - disabling calling hook `_delete' on `Ipinfo_Module'
DEBUG : 0.00002: Ipinfo_Module -> _delete
DEBUG : Service config `dns' disabled for site - disabling calling hook `_delete' on `Dns_Module'
DEBUG : 0.00001: Billing_Module -> _delete
DEBUG : 0.00001: Site_Module -> _delete
WARNING: Email_Module::remove_virtual_transport(): DNS is not configured for `daskjldfsdfs.com' - unable to remove MX records automatically
DEBUG : Deleting historical metrics
DEBUG : Firing *.end
DEBUG : Deleted site3, daskjldfsdfs.com (0) Succeeded
Interesting. Run upcp -sbfvv apnscp/assert-account-works
. This will give you the AddDomain command to reproduce, which is pretty similar to this with a few changes:
- dns,enabled=0
- dns,provider=null
- mail,enabled=0
- mail,provider=null
- ssl,enabled=0
- apache,jail=1 unless has_low_memory is set
You should be able to run that command and then try env DEBUG=1 DeleteDomain TESTDOMAIN
to get a better idea of what’s making a HTTP request and failing SSL check.
The first command fails at this step, however, the second one finishes without error.
TASK [apnscp/assert-account-works : Remove test account] ***************************************************************
task path: /usr/local/apnscp/resources/playbooks/roles/apnscp/assert-account-works/tasks/main.yml:49
fatal: [localhost]: FAILED! => {
"changed": false,
"cmd": [
"/usr/local/apnscp/bin/DeleteDomain",
"apiscp-int-vudaefluhghwstejrntxjlou.test"
],
"delta": "0:00:01.712987",
"end": "2024-12-07 01:20:00.045029",
"rc": 1,
"start": "2024-12-07 01:19:58.332042"
}
STDERR:
ERROR : unknown(): failed to remove site `site3': fatal(): fatal error retrieving URL: `Curl error: SSL certificate problem: certificate has expired'
0. Error_Reporter::add_error("failed to remove site `%s': %s", ["site3", "fatal(): fatal error retrieving URL: `Curl error: SSL certificate problem: certificate has expired'"])
[/usr/local/apnscp/lib/log_wrapper.php:72]
1. error("failed to remove site `%s': %s", "site3", "fatal(): fatal error retrieving URL: `Curl error: SSL certificate problem: certificate has expired'")
[/usr/local/apnscp/bin/DeleteDomain:99]
MSG:
non-zero return code
[root@s1 ~]# env DEBUG=1 DeleteDomain site3
DEBUG : Running hooks for `apiscp-int-vudaefluhghwstejrntxjlou.test' (user: `dhwurkjbmxbw')
DEBUG : 0.00065: Webapp_Module -> _delete
DEBUG : 0.00002: Sql_Module -> _delete
DEBUG : 0.00142: Php_Module -> _delete
DEBUG : 0.00029: Letsencrypt_Module -> _delete
DEBUG : 0.00002: Dav_Module -> _delete
DEBUG : 0.00002: Spamfilter_Module -> _delete
DEBUG : 0.00106: Majordomo_Module -> _delete
DEBUG : Service config `crontab' disabled for site - disabling calling hook `_delete' on `Crontab_Module'
DEBUG : Service config `vacation' disabled for site - disabling calling hook `_delete' on `Email_Module'
DEBUG : 0.00001: Ssh_Module -> _delete
DEBUG : Service config `mail' disabled for site - disabling calling hook `_delete' on `Email_Module'
DEBUG : Service config `pgsql' disabled for site - disabling calling hook `_delete' on `Pgsql_Module'
DEBUG : 0.00001: Ftp_Module -> _delete
DEBUG : 2.17089: Auth_Module -> _delete
DEBUG : 0.00003: Aliases_Module -> _delete
DEBUG : 0.00092: User_Module -> _delete
DEBUG : Service config `tomcat' disabled for site - disabling calling hook `_delete' on `Tomcat_Module'
DEBUG : Service config `ssl' disabled for site - disabling calling hook `_delete' on `Ssl_Module'
DEBUG : 0.00053: Bandwidth_Module -> _delete
DEBUG : 0.00002: Web_Module -> _delete
DEBUG : 0.06517: Mysql_Module -> _delete
DEBUG : 0.00003: Cgroup_Module -> _delete
DEBUG : Service config `ipinfo6' disabled for site - disabling calling hook `_delete' on `Ipinfo_Module'
DEBUG : 0.00002: Ipinfo_Module -> _delete
DEBUG : Service config `dns' disabled for site - disabling calling hook `_delete' on `Dns_Module'
DEBUG : 0.00002: Billing_Module -> _delete
DEBUG : 0.00001: Site_Module -> _delete
DEBUG : Deleting historical metrics
DEBUG : Firing *.end
DEBUG : Deleted site3, apiscp-int-vudaefluhghwstejrntxjlou.test (0) Succeeded
I think I found the issue. The ssl cert of cp proxy is ok but the one of cp api is expired and wasn’t renewed automatically for some reason.
Great. The problem was the expired certificate of cp-api. Now there’s just one account stuck:
[root@s1 virtual]# DeleteDomain --force site1
WARNING: Opcenter\Provisioning\Pam::disable(): pam service `ssh' not enabled
WARNING: Opcenter\Provisioning\Pam::disable(): pam service `ftp' not enabled
ERROR : Opcenter\Filesystem\Quota\Project::remove(): Specified path not found in projects
0. Error_Reporter::add_error("Specified path not found in projects", )
[/usr/local/apnscp/lib/log_wrapper.php:72]
1. error("Specified path not found in projects")
[/usr/local/apnscp/lib/Opcenter/Filesystem/Quota/Project.php:326]
2. Opcenter\Filesystem\Quota\Project->remove("/home/virtual/site1/shadow")
[/usr/local/apnscp/lib/Opcenter/Service/Validators/Diskquota/Group.php:74]
3. Opcenter\Service\Validators\Diskquota\Group->reconfigure("DEFAULT", null, Opcenter\SiteConfiguration)
[/usr/local/apnscp/lib/Opcenter/Service/Validators/Diskquota/Group.php:84]
4. Opcenter\Service\Validators\Diskquota\Group->depopulate(Opcenter\SiteConfiguration)
[/usr/local/apnscp/lib/Opcenter/Account/Delete.php:283]
5. Opcenter\Account\Delete->exec()
[/usr/local/apnscp/bin/DeleteDomain:97]
ERROR : Opcenter\Account\Delete::exec(): failed to depopulate service `diskquota' on svc var `group'
0. Error_Reporter::add_error("failed to depopulate service `%s' on svc var `%s'", ["diskquota", "group"])
[/usr/local/apnscp/lib/log_wrapper.php:72]
1. error("failed to depopulate service `%s' on svc var `%s'", "diskquota", "group")
[/usr/local/apnscp/lib/Opcenter/Account/Delete.php:284]
2. Opcenter\Account\Delete->exec()
[/usr/local/apnscp/bin/DeleteDomain:97]
After executing /usr/local/apnscp/bin/scripts/mapCheck.php rebuild
again, the error changes to this:
[root@s1 virtual]# DeleteDomain --force site1
WARNING: Undefined array key "ipinfo"
[/usr/local/apnscp/lib/modules/email.php:1790]
0. Email_Module->_delete()
[/usr/local/apnscp/lib/Util/Account/Hooks.php:140]
1. Util_Account_Hooks->_process("delete", )
[/usr/local/apnscp/lib/Util/Account/Hooks.php:50]
2. Util_Account_Hooks->run("delete")
[/usr/local/apnscp/lib/Opcenter/Account/Delete.php:207]
3. Opcenter\Account\Delete->processHooks()
[/usr/local/apnscp/lib/Opcenter/Account/Delete.php:238]
4. Opcenter\Account\Delete->exec()
[/usr/local/apnscp/bin/DeleteDomain:97]
WARNING: Undefined array key "ipinfo"
[/usr/local/apnscp/lib/modules/email.php:1790]
0. Email_Module->_delete()
[/usr/local/apnscp/lib/Util/Account/Hooks.php:140]
1. Util_Account_Hooks->_process("delete", )
[/usr/local/apnscp/lib/Util/Account/Hooks.php:50]
2. Util_Account_Hooks->run("delete")
[/usr/local/apnscp/lib/Opcenter/Account/Delete.php:207]
3. Opcenter\Account\Delete->processHooks()
[/usr/local/apnscp/lib/Opcenter/Account/Delete.php:238]
4. Opcenter\Account\Delete->exec()
[/usr/local/apnscp/bin/DeleteDomain:97]
WARNING: Opcenter\Provisioning\Pam::disable(): pam service `ssh' not enabled
WARNING: Opcenter\Provisioning\Pam::disable(): pam service `ftp' not enabled
ERROR : Opcenter\Filesystem\Quota\Project::remove(): Specified path not found in projects
0. Error_Reporter::add_error("Specified path not found in projects", )
[/usr/local/apnscp/lib/log_wrapper.php:72]
1. error("Specified path not found in projects")
[/usr/local/apnscp/lib/Opcenter/Filesystem/Quota/Project.php:326]
2. Opcenter\Filesystem\Quota\Project->remove("/home/virtual/site1/shadow")
[/usr/local/apnscp/lib/Opcenter/Service/Validators/Diskquota/Group.php:74]
3. Opcenter\Service\Validators\Diskquota\Group->reconfigure("DEFAULT", null, Opcenter\SiteConfiguration)
[/usr/local/apnscp/lib/Opcenter/Service/Validators/Diskquota/Group.php:84]
4. Opcenter\Service\Validators\Diskquota\Group->depopulate(Opcenter\SiteConfiguration)
[/usr/local/apnscp/lib/Opcenter/Account/Delete.php:283]
5. Opcenter\Account\Delete->exec()
[/usr/local/apnscp/bin/DeleteDomain:97]
ERROR : Opcenter\Account\Delete::exec(): failed to depopulate service `diskquota' on svc var `group'
0. Error_Reporter::add_error("failed to depopulate service `%s' on svc var `%s'", ["diskquota", "group"])
[/usr/local/apnscp/lib/log_wrapper.php:72]
1. error("failed to depopulate service `%s' on svc var `%s'", "diskquota", "group")
[/usr/local/apnscp/lib/Opcenter/Account/Delete.php:284]
2. Opcenter\Account\Delete->exec()
[/usr/local/apnscp/bin/DeleteDomain:97]
Ohh and I noticed another idling server which also had hundrets of artificial accounts stuck. The same steps could resolve it. Apparently there’s a problem in this test account routine when something goes wrong, like an expired certificate which can happen from time to time, and then the server is spammed.
Looks like you have a project quota setting for the account (“group” directive in siteXX/info/current/diskquota
) but no corresponding map in /etc/projid. Edit that file directly, set the line to “group=None” then try deleting again.
Thanks a lot. Now everything appears to be working again. 