New Domain Creation Fails: Internal Server Error from PowerDNS

Support
1

Description

When attempting to create a new site, both via WHMCS automation and manually, the following error is being given:
ERROR : Opcenter\Service\Validators\Common\Enabled::Opcenter\Service\Validators\Common{closure}(): failed to commit service configuration for dns': Server error: GET http://localhost:8081/api/v1/servers/localhost/zones/DOMAIN.com` resulted in a 500 Internal Server Error response:
{“error”: "Could not retrieve Domain Info: GSQLBackend unable to retrieve information about domain ‘DOMAIN.com’: Could (truncated…)

Steps to Reproduce

Please provide detailed steps to reproduce the bug. Include any necessary code snippets, commands, or configuration files.
Create a site via Nexus.

Expected Behavior

Site is created successfully

Actual Behavior

Seems there is an issue with PowerDNS, but I’m not exactly sure what’s causing this all of a sudden.

Environment

ApisCP version:

revision: 8d431715f1e49ad2fe5ad91087d748ac87e58ca7
timestamp: 1778257269
ver_maj: 3
ver_min: 2
ver_patch: 48
ver_pre: ''
dirty: false
debug: false

Operating System: 4.18.0-553.109.1.el8_10.x86_64

Additional relevant information (e.g., PHP version, database, etc.):
MariaDB Verion: Server version: 10.11.16-MariaDB MariaDB Server
PowerDNS Version: PowerDNS Authoritative Server 4.8.5 (C) 2001-2022 PowerDNS.COM BV

2

I was also able to get the same error when trying to curl from the localhost.

3

I’d approach this by looking at the pdns service with journalctl -n50 -u pdns. It’s hard to discern an error without logging; the error is generated by PowerDNS, so you need to look there first.

systemctl status pdns would be the second logical step if the log is “clean”.

4

That was the first thing I checked. PDNS is running and no errors are being logged aside from the first curl attempt without the API key and the outdated version message.

5

I can’t help without diagnostics. There’s something in /var/log/messages or systemctl or journalctl.

6

This is all that’s in /var/log/messages prior to me messing with the PDNS version thinking it needed updated (updating broke things more, so rolled it back to 4.8, but the initial issue persisted):

May 12 03:23:33 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 03:53:33 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 04:23:33 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 04:53:33 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 05:23:33 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 05:53:34 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 06:23:34 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 06:53:34 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 07:23:34 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 07:53:34 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 08:23:34 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 08:53:34 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 09:23:34 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 09:53:35 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 10:23:35 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 10:53:35 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 11:23:35 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 11:53:35 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 12:23:35 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 12:53:35 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 13:23:36 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 13:53:36 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 14:23:36 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 14:53:36 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 15:23:36 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 15:53:36 s1 pdns_server[4075255]: PowerDNS Security Update Mandatory: Unsupported release (EOL)
May 12 16:02:00 s1 pdns_server[4075255]: [webserver] REDACTED HTTP Request "/api/v1/servers/localhost/zones/DOMAIN.com": Authentication by API Key failed

That last request was me curl’ing the API without the API key. There is zero log of the requests from Apis, or my authenticated request of that domain.

7

I’ve even just tried increasing the log level to 3 (default 4), and webserver_log level from normal to detailed, nothing.

8

Some more info: It doesn’t matter the domain, any query over the API to /api/v1/servers/localhost/zones/DOMAIN HERE results in that 500 Internal Server error, definetly a powerDNS issue. Everything I’m finding points to something in the mysql database backend being wrong enough that powerdns dumps the request. Normally it tells you what failed, but that doesn’t seem to be the case here, I’ve increased logging to 9, and I can see the webserver request coming in, failing, but still only giving the 500 error on the zone in question. Is there a clean way to go about wiping that database and re-loading the info on it via ApisCP?

9

On my PowerDNS server, I have a Galera Cluster running on Ubuntu. Officially unsupported for ApisCP but it works.
I have nginx as a proxy in front of my PDNS API and it logs the 500 errors like this:

/var/log/nginx/dns1.dnsdomain.net.access.log.8.gz:[HOST IP] - - [05/May/2026:00:41:35 +0000] "GET /api/v1/servers/localhost/zones/customerdomain.com. HTTP/1.1" 500 34 "-" "ApisCP 3.2"

That’s a start, typically if I saw a 500 error it would work the next time so it was temporary.

10

Yeah that’s about all I’m getting right now as well. But it hasn’t been temporary, so far it’s persisted on both domains that the cluster has and doesn’t have, and has been like this probably since before yesterday, but that’s when I noticed it.

12

Set webserver-loglevel to detailed?

And check the credentials /etc/pdns/pdns.conf are correct (upcp -sb software/powerdns would assert that). Maybe /var/log/mysqld.log as well. Could be a corrupted table causing the query to fail. pdnsutil check-zone as well.

13

Since I’ve made this thread I’ve re-ran upcp -sb software/powerdns so the credentials should be correct, otherwise I would be seeing the failed authentication log entries each time I tried to add a domain from the panel.
Nothing of relevance in the mysql log, only
2026-05-13 17:50:31 1287465 [Warning] Aborted connection 1287465 to db: 'powerdns' user: 'powerdns' host: 'localhost' (Got an error reading communication packets) and that only happened on pdns startup, that log entry doesn’t appear when I try to curl the API.

I also ran check-zone on all zones (gathered by running list-all-zones), and no errors on any of them.