Hi,
On three ApisCP servers (two EL8, one EL9; all on v3.2.49.2 at bdc0ff858, panel Apache/2.4.67, panel PHP 8.3.36) the panel frontend is restarted by apnscpd’s own watchdog several times a day. It recovers by itself, so this is not an outage report, but I’d like to understand why the frontend stops answering.
What the logs show, each time:
storage/logs/start.log:
WARNING: ListenerService\Daemon::checkFrontend(): Panel found unresponsive - restarting
INFO : Frontend HTTP started with PID `NNNN’
storage/logs/error_log (panel httpd):
[core:error] AH00046: child process NNNN still did not exit, sending a SIGKILL
[mpm_event:notice] AH00491: caught SIGTERM, shutting down
… ~50 s later …
[mpm_event:notice] AH00489: Apache/2.4.67 (apnscp) … resuming normal operations
During those ~50 s, :2083 refuses connections and monit’s :2082 check fails (“Connection refused”).
Frequency (count of AH00046 SIGKILL lines per weekly panel error_log, since late September):
server A (EL9): 143 / 174 / 164 / 127 → about 20 per day
server B (EL8): 36 / 33 / 16 / 28 → about 4 per day
server C (EL8): 42 / 30 / 32 / 40 → about 5 per day
It does not look load-related. Before one restart the panel access_log shows only monitoring requests (Monit, an external uptime check) and the “apnscp Internal check” HEAD / and HEAD /apps/login once a minute. The last internal check answered at 13:09:55, the next one never appears, and the SIGKILL follows at 13:11:21. Panel php-fpm: pm=ondemand, max_children=10, request_terminate_timeout=0. Nothing in php-fpm.log around those times.
This is different from “Apiscp become unresponsive after Verifying license” (#970) and from the stuck monit event in #818: here the hang is short and the watchdog recovers it.
Questions:
- Is a few restarts a day expected behaviour of checkFrontend, or a sign that something in the panel hangs?
- What does checkFrontend consider “unresponsive” (timeout, number of misses)?
- What would you like captured next time? I can run an strace or a server-status snapshot on the panel httpd when the internal check stops answering, if you tell me which is useful.
Thanks,
Bogdan (with Claude)