Failed to start ApisCP backend service - after trying to upgrade from older version.

Bug Report Template

Description

I’ve tried to upgrade from an older version to current. “Failed to start ApisCP backend service” during this process. Have proceeded to manually upgrade through 3.2.38 (by "git checkout’ each version to try to get to a stable place). I’m now stuck trying to get upgraded further. “upcp” fails. and if I try to pull 3.2.39.x or latest, I see different failures based on which version I try.

Steps to Reproduce

upcp
OR
./artisan migrate --force

Expected Behavior

Successful upgrade to latest version

Actual Behavior

upcp results in:

Please commit your changes or stash them before you switch branches.
error: The following untracked working tree files would be overwritten by checkout:
	docs/admin/Authentication.md
	lib/Module/Support/Webapps/VersionFetcher/PackageJson.php
	lib/Opcenter/Crypto/Keyring.php
	lib/Opcenter/Crypto/KeyringTrait.php
	lib/Opcenter/Crypto/SymmetricCrypt.php
	lib/Opcenter/Service/Validators/Cgroup/Delegator.php
	lib/Opcenter/System/Cgroup/Hierarchy.php
Please move or remove them before you switch branches.
Aborting
PHP Parse error:  syntax error, unexpected '->' (T_OBJECT_OPERATOR) in /usr/local/apnscp/lib/error_reporter.php on line 705
Job for apnscp.service failed because the control process exited with error code.
See "systemctl status apnscp.service" and "journalctl -xe" for details.
PHP Parse error:  syntax error, unexpected '->' (T_OBJECT_OPERATOR) in /usr/local/apnscp/lib/error_reporter.php on line 705
Upgrade requested, but old = new (b30a6a04b5eab1133a6fa2095ab7d16d0f223df7)
./artisan migrate --force

results in…

PHP Parse error:  syntax error, unexpected '->' (T_OBJECT_OPERATOR) in /usr/local/apnscp/lib/error_reporter.php on line 705

Environment

ApisCP version: cpcmd misc:cp-version
error: PHP Parse error: syntax error, unexpected ‘->’ (T_OBJECT_OPERATOR) in /usr/local/apnscp/lib/error_reporter.php on line 705

But, on version 3.2.38.2.

Operating System:
4.18.0-348.7.1.el8_5.x86_64

Additional relevant information (e.g., PHP version, database, etc.):
N/A

Additional Information

Hosted websites still functional. Admin panel not accessible.

Performing a leap like this disregards any incremental prerequisite migrations, including database schema or platform features, that present code relies upon¹.

Jump back to 3.2.38.2, then run upcp to perform a stepwise migration, which will apply pending migrations at each upgrade. If there’s any errors during that migration phase, then let me know as they will need to be addressed before proceeding further.

git reset --hard
git checkout 3.2.38.2
upcp --reset
upcp

1: upcp -sbf, followed by ./artisan migrate may also work but YMMV.

Thanks for the quick response!

errors on the “git checkout v3.2.38.2”…

[root@v1 apnscp]# git reset --hard
HEAD is now at b30a6a04b FIX: extraneous indention (Webapps)
[root@v1 apnscp]# git checkout 3.2.38.2
error: pathspec '3.2.38.2' did not match any file(s) known to git
[root@v1 apnscp]# git checkout v3.2.38.2
error: The following untracked working tree files would be overwritten by checkout:
	resources/playbooks/roles/packages/install/templates/mariadb.repo.j2
	resources/playbooks/roles/packages/install/templates/mongodb.repo.j2
	resources/storehouse/migration/jsonpickle/images/jsonpickle-0.9.6.svg
	resources/storehouse/migration/jsonpickle/images/jsonpickle-1.5.1.svg
	resources/storehouse/migration/jsonpickle/images/jsonpickle-1.5.2.svg
	resources/storehouse/migration/jsonpickle/images/jsonpickle-2.2.0.svg
	resources/storehouse/migration/jsonpickle/images/jsonpickle-3.0.0-dev.svg
Please move or remove them before you switch branches.
Aborting
git reset --hard v3.2.38.2
git stash save old-code
# Alternatively run "git clean -di" to delete all untracked files
# "git status" will show untracked
upcp --reset
upcp
[root@v1 apnscp]# git reset --hard v3.2.38.2
HEAD is now at fc84cec46 CHG: expose cert_exists to admin (ssl) CHG: prep for 3.2.38.2 (migrations)
[root@v1 apnscp]# git stash save old-code
No local changes to save
[root@v1 apnscp]# git clean -di
[root@v1 apnscp]# git status
HEAD detached from v3.2.44
nothing to commit, working tree clean
[root@v1 apnscp]# upcp --reset
warning: unable to unlink 'resources/playbooks/roles/packages/install/templates/mariadb.repo.j2': Permission denied
warning: unable to unlink 'resources/playbooks/roles/packages/install/templates/mongodb.repo.j2': Permission denied
warning: unable to unlink 'resources/storehouse/migration/jsonpickle/images/jsonpickle-0.9.6.svg': Permission denied
warning: unable to unlink 'resources/storehouse/migration/jsonpickle/images/jsonpickle-1.5.1.svg': Permission denied
warning: unable to unlink 'resources/storehouse/migration/jsonpickle/images/jsonpickle-1.5.2.svg': Permission denied
warning: unable to unlink 'resources/storehouse/migration/jsonpickle/images/jsonpickle-2.2.0.svg': Permission denied
warning: unable to unlink 'resources/storehouse/migration/jsonpickle/images/jsonpickle-3.0.0-dev.svg': Permission denied
error: unable to create file resources/storehouse/migration/jsonpickle/images/jsonpickle-0.9.6-3.11.svg: Permission denied
error: unable to create file resources/storehouse/migration/jsonpickle/images/jsonpickle-0.9.6-3.9.svg: Permission denied
error: unable to create file resources/storehouse/migration/jsonpickle/images/jsonpickle-1.5.1-3.11.svg: Permission denied
error: unable to create file resources/storehouse/migration/jsonpickle/images/jsonpickle-1.5.1-3.9.svg: Permission denied
error: unable to create file resources/storehouse/migration/jsonpickle/images/jsonpickle-1.5.2-3.11.svg: Permission denied
error: unable to create file resources/storehouse/migration/jsonpickle/images/jsonpickle-1.5.2-3.9.svg: Permission denied
error: unable to create file resources/storehouse/migration/jsonpickle/images/jsonpickle-2.2.0-3.11.svg: Permission denied
error: unable to create file resources/storehouse/migration/jsonpickle/images/jsonpickle-2.2.0-3.9.svg: Permission denied
error: unable to create file resources/storehouse/migration/jsonpickle/images/jsonpickle-3.0.0-3.11.svg: Permission denied
error: unable to create file resources/storehouse/migration/jsonpickle/images/jsonpickle-3.0.0-3.9.svg: Permission denied
fatal: Could not reset index file to revision 'origin/master'.
Reasserting permissions
resources/storehouse/migration/jsonpickle/images
resources/storehouse/migration/jsonpickle/images/jsonpickle-0.9.6.svg
resources/storehouse/migration/jsonpickle/images/jsonpickle-1.5.1.svg
resources/storehouse/migration/jsonpickle/images/jsonpickle-3.0.0-dev.svg
resources/storehouse/migration/jsonpickle/images/jsonpickle-2.2.0.svg
resources/storehouse/migration/jsonpickle/images/jsonpickle-1.5.2.svg
resources/playbooks/roles/php/install-pecl-module
resources/playbooks/roles/packages/install/templates
resources/playbooks/roles/packages/install/templates/mongodb.repo.j2
resources/playbooks/roles/packages/install/templates/mariadb.repo.j2
storage/tmp/agent-root
.git/index
.git/ORIG_HEAD
.git/HEAD
HEAD is now at 48b6fd0a4 CHG: add R10, R11 fingerprints (Letsencrypt) CHG: elevate fingerprint mismatch to warning (Letsencrypt)
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Nothing to install, update or remove
Package swiftmailer/swiftmailer is abandoned, you should avoid using it. Use symfony/mailer instead.
Package symfony/debug is abandoned, you should avoid using it. Use symfony/error-handler instead.
Package true/punycode is abandoned, you should avoid using it. No replacement was suggested.
Generating autoload files
composer/package-versions-deprecated: Generating version class...
composer/package-versions-deprecated: ...done generating version class
90 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
resources/playbooks/roles/php/install-pecl-module
PHP Parse error:  syntax error, unexpected '|', expecting '{' in /usr/local/apnscp/lib/helpers.php on line 181

upcp --reset moves HEAD to master, which is the problem. This should do it:

# Move pointer to before v3.2.38.2 release
git reset --hard v3.2.38.2^
# Verify your update policy is major:
cpcmd scope:get cp.update-policy
# If not, cpcmd scope:set cp.update-policy major
# Alternatively, to junk all changes: git clean -df 
git stash save old-code
# "git describe" will report v3.2.38.1-2
# upcp runs as user apnscp, so let's make sure it has permission
chown -R apnscp:apnscp /usr/local/apnscp
# Finally, move through migrations
upcp

Your migrations may fail, and if so, let me know on the context. We’ll go from there.

So, this fails with the error I’ve seen repeated throughout this process…

[root@v1 apnscp]# cpcmd scope:get cp.update-policy
PHP Parse error:  syntax error, unexpected '|', expecting '{' in /usr/local/apnscp/lib/helpers.php on line 181

This is the code in /usr/local/apnscp/lib/helpers.php at line 181:

	if (!function_exists('openssl_cipher_key_length')) {
                // @TODO remove in PHP 8.2+
                function openssl_cipher_key_length(string $cipher_algo): int|false
                {
                        $length = match (strtolower($cipher_algo)) {
                                'aes-128-cbc' => 16,
                                'aes-128-cbc-hmac-sha1' => 16,
                                'aes-128-cbc-hmac-sha256' => 16,
                                ...

PHP 8.1 was introduced in 3.2.37 through a migration and that syntax error is due to union types, so you’re still using PHP 7.4.

Give this a shot:

git checkout v3.2.37
systemctl restart apiscp
./artisan migrate --force

Then confirm panel PHP is using PHP 8.1:

apnscp_php -v
# You'll see something similar to:
# PHP 8.1.21 (cli) (built: Aug  1 2023 02:43:07) (NTS)

Once that’s confirmed, cpcmd scope:get cp.update-policy should work as intended. Run the steps outlined above to replay the migration backlog.

git checkout v3.2.37
systemctl restart apiscp

results in:

[root@v1 apnscp]# systemctl restart apiscp
Job for apnscp.service failed because the control process exited with error code.
See "systemctl status apnscp.service" and "journalctl -xe" for details.

I did run the ./artisan migrate --force and it also failed, here:

[1/1] TASK [mail/configure-postfix : Setup cyrus] ************************************
included: /usr/local/apnscp/resources/playbooks/roles/mail/configure-postfix/tasks/setup-cyrus.yml for localhost
[1/1] TASK [mail/configure-postfix : Verify saslauthd installed] *********************
fatal: [localhost]: FAILED! => changed=false 
  msg: 'Failed to download metadata for repo ''appstream'': Cannot prepare internal mirrorlist: Curl error (6): Couldn''t resolve host name for http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=AppStream&infra=stock [Could not resolve host: mirrorlist.centos.org]'
  rc: 1
  results: []
[1/1] PLAY RECAP *********************************************************************
localhost                  : ok=9    changed=0    unreachable=0    failed=1    skipped=3    rescued=0    ignored=0
(RuntimeException) EXCEPTION: STDOUT: 
PLAY [localhost] ***************************************************************

note: I’ve tried to ping mirrorlist.centos.org from that server and from a couple of other machines that use different DNS. None of them can resolve “mirrorlist.centos.org”.

obviously, apnscp_php -v is still showing the old php version v.7.4.27

appreciate your help with this!!

Just realized that you may want this…

[root@v1 apnscp]# systemctl status apnscp.service
● apnscp.service - ApisCP backend service
   Loaded: loaded (/etc/systemd/system/apnscp.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/apnscp.service.d
           └─override.conf
   Active: failed (Result: exit-code) since Mon 2024-08-12 11:36:10 CDT; 17s ago
  Process: 73636 ExecStopPost=/usr/sbin/httpd -f /usr/local/apnscp/config/httpd.conf -k stop $HTTPD_ARGS (code=exited, status=0/SUCCESS)
  Process: 73627 ExecStart=/usr/local/apnscp/bin/apnscpd start (code=exited, status=255)
 Main PID: 73627 (code=exited, status=255)
   Status: "Detaching"
    Tasks: 4 (limit: 23713)
   Memory: 47.4M
   CGroup: /system.slice/apnscp.service
           └─73131 redis-server 127.0.0.1:0

Aug 12 11:36:10 v1.rupp.cloud systemd[1]: apnscp.service: Found left-over process 73131 (redis-server) in control group while starting unit. Ignoring.
Aug 12 11:36:10 v1.rupp.cloud systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Aug 12 11:36:10 v1.rupp.cloud systemd[1]: Starting ApisCP backend service...
Aug 12 11:36:10 v1.rupp.cloud systemd[1]: apnscp.service: Supervising process 73627 which is not our child. We'll most likely not notice when it exits.
Aug 12 11:36:10 v1.rupp.cloud systemd[1]: apnscp.service: Main process exited, code=exited, status=255/n/a
Aug 12 11:36:10 v1.rupp.cloud httpd[73636]: httpd (no pid file) not running
Aug 12 11:36:10 v1.rupp.cloud systemd[1]: apnscp.service: Failed with result 'exit-code'.
Aug 12 11:36:10 v1.rupp.cloud systemd[1]: Failed to start ApisCP backend service.

And nothing unique in “journalctl -xe” aside from failed logins.

What’s in /etc/resolv.conf? If 127.0.0.53, is systemd-resolved service running? What’s in /etc/redhat-release?

here you go…

[root@v1 ~]# more /etc/resolv.conf
# Generated by NetworkManager
nameserver 1.0.0.1
nameserver 1.1.1.1

[root@v1 ~]# ping 1.0.0.1
PING 1.0.0.1 (1.0.0.1) 56(84) bytes of data.
64 bytes from 1.0.0.1: icmp_seq=1 ttl=58 time=2.09 ms
64 bytes from 1.0.0.1: icmp_seq=2 ttl=58 time=2.09 ms
^C
--- 1.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 2.088/2.090/2.093/0.045 ms

[root@v1 ~]# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=58 time=1.64 ms
^C
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.639/1.639/1.639/0.000 ms

[root@v1 ~]# ping www.google.com
PING www.google.com (142.250.190.132) 56(84) bytes of data.
64 bytes from ord37s36-in-f4.1e100.net (142.250.190.132): icmp_seq=1 ttl=117 time=1.42 ms
64 bytes from ord37s36-in-f4.1e100.net (142.250.190.132): icmp_seq=2 ttl=117 time=1.35 ms
64 bytes from ord37s36-in-f4.1e100.net (142.250.190.132): icmp_seq=3 ttl=117 time=1.35 ms
^C
--- www.google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.349/1.372/1.420/0.054 ms

[root@v1 ~]# ping mirrorlist.centos.org
ping: mirrorlist.centos.org: Name or service not known
[root@v1 ~]# ping centos.org
PING centos.org (81.171.33.202) 56(84) bytes of data.
64 bytes from ip-81.171.33.202.centos.org (81.171.33.202): icmp_seq=1 ttl=51 time=104 ms
64 bytes from ip-81.171.33.202.centos.org (81.171.33.202): icmp_seq=2 ttl=51 time=104 ms
64 bytes from ip-81.171.33.202.centos.org (81.171.33.202): icmp_seq=3 ttl=51 time=104 ms
^C
--- centos.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 103.955/103.987/104.025/0.373 ms

[root@v1 ~]# systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients

[root@v1 ~]# more /etc/redhat-release
CentOS Linux release 8.5.2111

[root@v1 ~]# systemctl enable systemd-resolved
Created symlink /etc/systemd/system/dbus-org.freedesktop.resolve1.service → /usr/lib/systemd/system/systemd-resolved.service.
Created symlink /etc/systemd/system/multi-user.target.wants/systemd-resolved.service → /usr/lib/systemd/system/systemd-resolved.service.

[root@v1 ~]# systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients

Doing some digging, it looks like the repositories should have been updated when my system upgraded to CentOS 8.x but weren’t. Some discussion on the topic here:

Should one of your migration playbooks have made this happen? i.e. do you have a preferred suggestion for me to fix this?

Thanks!

CentOS 8 became Stream as of January 2022. Moreover, it looks like Stream 8 went EOL May 31. Going forward, you’d need to perform an in-place migration to Rocky Linux or AlmaLinux, migrating to Stream might work but - again, it’s EOL so no further security updates are delivered.

Change the repo URL from mirror.centos.org to vault.centos.org for all matching repos (ref: StackExchange):

sed -i s/mirror.centos.org/vault.centos.org/g /etc/yum.repos.d/*.repo
sed -i s/^#.*baseurl=http/baseurl=http/g /etc/yum.repos.d/*.repo
sed -i s/^mirrorlist=http/#mirrorlist=http/g /etc/yum.repos.d/*.repo

At this point, dnf should work once again allowing upcp migrations to complete. You can switch OS distros using cpcmd scope:set system.distro rocky (or “alma” for AlmaLinux) once everything is in proper working order. Rocky Linux and AlmaLinux will continue to be supported until May 2029.

dnf fixed with the suggestion above.

apnscp is still failing to start/restart (I’ve tried a reboot and “systemctl start/restart apnscp”):

[root@v1 apnscp]# systemctl status apnscp
● apnscp.service - ApisCP backend service
   Loaded: loaded (/etc/systemd/system/apnscp.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/apnscp.service.d
           └─override.conf
   Active: failed (Result: exit-code) since Mon 2024-08-12 15:33:21 CDT; 1min 41s ago
  Process: 117911 ExecStopPost=/usr/sbin/httpd -f /usr/local/apnscp/config/httpd.conf -k stop $HTTPD_ARGS (code=exited, status=0/SUCCESS)
  Process: 117902 ExecStart=/usr/local/apnscp/bin/apnscpd start (code=exited, status=255)
 Main PID: 117902 (code=exited, status=255)
   Status: "Detaching"
    Tasks: 4 (limit: 23713)
   Memory: 56.0M
   CGroup: /system.slice/apnscp.service
           └─115027 redis-server 127.0.0.1:0

Aug 12 15:33:21 v1.rupp.cloud systemd[1]: Starting ApisCP backend service...
Aug 12 15:33:21 v1.rupp.cloud systemd[1]: apnscp.service: Supervising process 117902 which is not our child. We'll most likely not notice when it exits.
Aug 12 15:33:21 v1.rupp.cloud systemd[1]: apnscp.service: Main process exited, code=exited, status=255/n/a
Aug 12 15:33:21 v1.rupp.cloud httpd[117911]: httpd (no pid file) not running
Aug 12 15:33:21 v1.rupp.cloud systemd[1]: apnscp.service: Failed with result 'exit-code'.
Aug 12 15:33:21 v1.rupp.cloud systemd[1]: Failed to start ApisCP backend service.

(I believe) as a result of apnscp not running, “./artisan migrate --force” now fails a number of tasks with the following be common on all of them:

[1/1] TASK [common : Filter accounts matching "{'cgroup': {'enabled': 1}}"] **********
fatal: [localhost]: FAILED! => changed=false 
  cmd:
  - /usr/local/apnscp/bin/cmd
  - -i
  - json
  - -o
  - json
  - admin:collect
  - '{"null": "ansible-dummy-field"}'
  - '{"cgroup": {"enabled": 1}}'
  delta: '0:00:00.230018'
  end: '2024-08-12 15:30:57.783541'
  msg: non-zero return code
  rc: 255
  start: '2024-08-12 15:30:57.553523'
  stderr: |-
    WARNING: socket_connect(): unable to connect [2]: No such file or directory
    [/usr/local/apnscp/lib/datastream.php:277]

I do have a backup from a few days ago that I’m tempted to restore (my websites are all down at this point) to at least get the service running again. At which point, I could try to upgrade again and/or migrate the sites to a new server (I have one installed that I could use).

Thoughts on the best path forward?

Backend isn’t starting. What’s the last 30 or so lines in /usr/local/apnscp/storage/logs/start.log? What does git describe report in /usr/local/apnscp?

Dozens of repeats of,

EXCEPTION: Class 'Dbus' not found 
[/usr/local/apnscp/lib/Opcenter/Dbus/Systemd.php:227]

         0B. Opcenter\Dbus\Systemd->proxy("org.freedesktop.systemd1.Manager")
            [/usr/local/apnscp/lib/Opcenter/Dbus/Systemd.php:46]
         1B. Opcenter\Dbus\Systemd->unitToObject("apiscp.service", true)
            [/usr/local/apnscp/lib/Opcenter/Dbus/Systemd.php:82]
         2B. Opcenter\Dbus\Systemd->exists("apiscp")
            [/usr/local/apnscp/lib/Opcenter/System/GenericSystemdService.php:177]
         3B. Opcenter\System\GenericSystemdService::exists()
            [/usr/local/apnscp/lib/Opcenter/System/GenericSystemdService.php:80]
         4B. Opcenter\System\GenericSystemdService::reset()
            [/usr/local/apnscp/lib/lservicelib.pht:204]
         5B. ListenerService\Daemon->start()
            [/usr/local/apnscp/lib/lservicelib.pht:169]
         6B. ListenerService\Daemon->__construct()
            [/usr/local/apnscp/lib/lservicelib.pht:1446]
         7B. ListenerService\Daemon::init()
            [/usr/local/apnscp/lib/lservice.php:28]
[Mon Aug 12 17:16:19 2024] EXCEPTION: Class 'Dbus' not found 
[/usr/local/apnscp/lib/Opcenter/Dbus/Systemd.php:227]


         0B. Error_Reporter::get_debug_bt()
            [/usr/local/apnscp/lib/error_reporter.php:280]
         1B. Error_Reporter::handle_error(128, "Class 'Dbus' not found", "/usr/local/apnscp/lib/Opcenter/Dbus/Systemd.php", 227, Error)
            [/usr/local/apnscp/lib/error_reporter.php:159]
         2B. Error_Reporter::handle_exception(Error)
            [n/a]

EXCEPTION: Class 'Dbus' not found 
[/usr/local/apnscp/lib/Opcenter/Dbus/Systemd.php:227]

         0B. Opcenter\Dbus\Systemd->proxy("org.freedesktop.systemd1.Manager")
            [/usr/local/apnscp/lib/Opcenter/Dbus/Systemd.php:46]
         1B. Opcenter\Dbus\Systemd->unitToObject("apiscp.service", true)
            [/usr/local/apnscp/lib/Opcenter/Dbus/Systemd.php:82]
         2B. Opcenter\Dbus\Systemd->exists("apiscp")
            [/usr/local/apnscp/lib/Opcenter/System/GenericSystemdService.php:177]
         3B. Opcenter\System\GenericSystemdService::exists()
            [/usr/local/apnscp/lib/Opcenter/System/GenericSystemdService.php:80]
         4B. Opcenter\System\GenericSystemdService::reset()
            [/usr/local/apnscp/lib/lservicelib.pht:204]
         5B. ListenerService\Daemon->start()
            [/usr/local/apnscp/lib/lservicelib.pht:169]
         6B. ListenerService\Daemon->__construct()
            [/usr/local/apnscp/lib/lservicelib.pht:1446]
         7B. ListenerService\Daemon::init()
            [/usr/local/apnscp/lib/lservice.php:28]

[root@v1 apnscp]# git describe 
v3.2.37

D-Bus was introduced in 3.2.36.1, so you’re even further back that initially thought.

upcp -sbf apnscp/install-extensions will build any missing extensions. Once done, systemctl restart apiscp. If it boots correctly, proceed with upcp.

upcp -sbf apnscp/install-extensions

failed…

TASK [php/install-pecl-module : Build and install dbus library.] ***********************************************************************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/usr/local/apnscp/resources/playbooks/roles/php/install-pecl-module/files/installPecl.sh", "https://github.com/apisnetworks/pecl-dbus.git", "master"], "delta": "0:00:02.475959", "end": "2024-08-12 19:34:27.233949", "msg": "non-zero return code", "rc": 1, "start": "2024-08-12 19:34:24.757990", "stderr": "Cloning into 'extension'...\nAlready on 'master'\nconfigure: WARNING: you should use --build, --host, --target\nconfigure: error: The DBUS-1 package was not detected on your system", "stderr_lines": ["Cloning into 'extension'...", "Already on 'master'", "configure: WARNING: you should use --build, --host, --target", "configure: error: The DBUS-1 package was not detected on your system"], "stdout": "/usr/local/apnscp/storage/tmp/extension.vk7NxPr /usr/local/apnscp/resources/playbooks\n/usr/local/apnscp/storage/tmp/extension.vk7NxPr/extension /usr/local/apnscp/storage/tmp/extension.vk7NxPr /usr/local/apnscp/resources/playbooks\nYour branch is up to date with 'origin/master'.\n/usr/local/apnscp/storage/tmp/extension.vk7NxPr /usr/local/apnscp/resources/playbooks\n/usr/local/apnscp/storage/tmp/extension.vk7NxPr/extension /usr/local/apnscp/storage/tmp/extension.vk7NxPr /usr/local/apnscp/resources/playbooks\nConfiguring for:\nPHP Api Version:         20190902\nZend Module Api No:      20190902\nZend Extension Api No:   320190902\nchecking for grep that handles long lines and -e... /usr/bin/grep\nchecking for egrep... /usr/bin/grep -E\nchecking for a sed that does not truncate output... /usr/bin/sed\nchecking for pkg-config... /usr/bin/pkg-config\nchecking pkg-config is at least version 0.9.0... yes\nchecking for cc... cc\nchecking whether the C compiler works... yes\nchecking for C compiler default output file name... a.out\nchecking for suffix of executables... \nchecking whether we are cross compiling... no\nchecking for suffix of object files... o\nchecking whether we are using the GNU C compiler... yes\nchecking whether cc accepts -g... yes\nchecking for cc option to accept ISO C89... none needed\nchecking how to run the C preprocessor... cc -E\nchecking for icc... no\nchecking for suncc... no\nchecking for system library directory... lib\nchecking if compiler supports -R... no\nchecking if compiler supports -Wl,-rpath,... yes\nchecking build system type... x86_64-pc-linux-gnu\nchecking host system type... x86_64-pc-linux-gnu\nchecking target system type... x86_64-pc-linux-gnu\nchecking for PHP prefix... /usr/local/apnscp\nchecking for PHP includes... -I/usr/local/apnscp/sys/php/include/php -I/usr/local/apnscp/sys/php/include/php/main -I/usr/local/apnscp/sys/php/include/php/TSRM -I/usr/local/apnscp/sys/php/include/php/Zend -I/usr/local/apnscp/sys/php/include/php/ext -I/usr/local/apnscp/sys/php/include/php/ext/date/lib\nchecking for PHP extension directory... /usr/local/apnscp/sys/php/20190902-zts\nchecking for PHP installed headers prefix... /usr/local/apnscp/sys/php/include/php\nchecking if debug is enabled... no\nchecking if zts is enabled... yes\nchecking for gawk... gawk\nchecking whether to enable dbus functions... yes, shared\nchecking libxml2 install dir... no\nchecking for pkg-config... found\nchecking for dbus... not found", "stdout_lines": ["/usr/local/apnscp/storage/tmp/extension.vk7NxPr /usr/local/apnscp/resources/playbooks", "/usr/local/apnscp/storage/tmp/extension.vk7NxPr/extension /usr/local/apnscp/storage/tmp/extension.vk7NxPr /usr/local/apnscp/resources/playbooks", "Your branch is up to date with 'origin/master'.", "/usr/local/apnscp/storage/tmp/extension.vk7NxPr /usr/local/apnscp/resources/playbooks", "/usr/local/apnscp/storage/tmp/extension.vk7NxPr/extension /usr/local/apnscp/storage/tmp/extension.vk7NxPr /usr/local/apnscp/resources/playbooks", "Configuring for:", "PHP Api Version:         20190902", "Zend Module Api No:      20190902", "Zend Extension Api No:   320190902", "checking for grep that handles long lines and -e... /usr/bin/grep", "checking for egrep... /usr/bin/grep -E", "checking for a sed that does not truncate output... /usr/bin/sed", "checking for pkg-config... /usr/bin/pkg-config", "checking pkg-config is at least version 0.9.0... yes", "checking for cc... cc", "checking whether the C compiler works... yes", "checking for C compiler default output file name... a.out", "checking for suffix of executables... ", "checking whether we are cross compiling... no", "checking for suffix of object files... o", "checking whether we are using the GNU C compiler... yes", "checking whether cc accepts -g... yes", "checking for cc option to accept ISO C89... none needed", "checking how to run the C preprocessor... cc -E", "checking for icc... no", "checking for suncc... no", "checking for system library directory... lib", "checking if compiler supports -R... no", "checking if compiler supports -Wl,-rpath,... yes", "checking build system type... x86_64-pc-linux-gnu", "checking host system type... x86_64-pc-linux-gnu", "checking target system type... x86_64-pc-linux-gnu", "checking for PHP prefix... /usr/local/apnscp", "checking for PHP includes... -I/usr/local/apnscp/sys/php/include/php -I/usr/local/apnscp/sys/php/include/php/main -I/usr/local/apnscp/sys/php/include/php/TSRM -I/usr/local/apnscp/sys/php/include/php/Zend -I/usr/local/apnscp/sys/php/include/php/ext -I/usr/local/apnscp/sys/php/include/php/ext/date/lib", "checking for PHP extension directory... /usr/local/apnscp/sys/php/20190902-zts", "checking for PHP installed headers prefix... /usr/local/apnscp/sys/php/include/php", "checking if debug is enabled... no", "checking if zts is enabled... yes", "checking for gawk... gawk", "checking whether to enable dbus functions... yes, shared", "checking libxml2 install dir... no", "checking for pkg-config... found", "checking for dbus... not found"]}

PLAY RECAP *****************************************************************************************************************************************************
localhost                  : ok=51   changed=3    unreachable=0    failed=1    skipped=15   rescued=0    ignored=0   

dnf install -y dbus-devel then try again.

… ran successfully.

But, I’ve tried the previous commands again with no success. The apiscp service still isn’t starting:

[root@v1 apnscp]# systemctl status apnscp.service 
● apnscp.service - ApisCP backend service
   Loaded: loaded (/etc/systemd/system/apnscp.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/apnscp.service.d
           └─override.conf
   Active: failed (Result: exit-code) since Tue 2024-08-13 20:56:52 CDT; 8s ago
  Process: 33902 ExecStopPost=/usr/sbin/httpd -f /usr/local/apnscp/config/httpd.conf -k stop $HTTPD_ARGS (code=exited, status=0/SUCCESS)
  Process: 33896 ExecStart=/usr/local/apnscp/bin/apnscpd start (code=exited, status=255)
 Main PID: 33896 (code=exited, status=255)
   Status: "Detaching"

Aug 13 20:56:52 v1.domain.tld apnscpd[33895]: PHP Warning:  PHP Startup: dbus: Unable to initialize module
Aug 13 20:56:52 v1.domain.tld apnscpd[33895]: Module compiled with build ID=API20190902,TS
Aug 13 20:56:52 v1.domain.tld apnscpd[33895]: PHP    compiled with build ID=API20190902,NTS
Aug 13 20:56:52 v1.domain.tld apnscpd[33895]: These options need to match
Aug 13 20:56:52 v1.domain.tld apnscpd[33895]:  in Unknown on line 0
Aug 13 20:56:52 v1.domain.tld systemd[1]: apnscp.service: Supervising process 33896 which is not our child. We'll most likely not notice when it exits.
Aug 13 20:56:52 v1.domain.tld systemd[1]: apnscp.service: Main process exited, code=exited, status=255/n/a
Aug 13 20:56:52 v1.domain.tld httpd[33902]: httpd (no pid file) not running
Aug 13 20:56:52 v1.domain.tld systemd[1]: apnscp.service: Failed with result 'exit-code'.
Aug 13 20:56:52 v1.domain.tld systemd[1]: Failed to start ApisCP backend service.

php still hasn’t upgraded past 7.4.33:

[root@v1 apnscp]# apnscp_php -v
Failed loading /usr/local/apnscp/sys/php/20190902-zts/opcache.so:  /usr/local/apnscp/sys/php/20190902-zts/opcache.so: undefined symbol: zend_signal_globals_offset
PHP Warning:  PHP Startup: Unable to load dynamic library 'apcu.so' (tried: /usr/local/apnscp/sys/php/20190902-zts/apcu.so (/usr/local/apnscp/sys/php/20190902-zts/apcu.so: undefined symbol: executor_globals_offset), /usr/local/apnscp/sys/php/20190902-zts/apcu.so.so (/usr/local/apnscp/sys/php/20190902-zts/apcu.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library 'igbinary.so' (tried: /usr/local/apnscp/sys/php/20190902-zts/igbinary.so (/usr/local/apnscp/sys/php/20190902-zts/igbinary.so: undefined symbol: core_globals_offset), /usr/local/apnscp/sys/php/20190902-zts/igbinary.so.so (/usr/local/apnscp/sys/php/20190902-zts/igbinary.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library 'redis.so' (tried: /usr/local/apnscp/sys/php/20190902-zts/redis.so (/usr/local/apnscp/sys/php/20190902-zts/redis.so: undefined symbol: executor_globals_offset), /usr/local/apnscp/sys/php/20190902-zts/redis.so.so (/usr/local/apnscp/sys/php/20190902-zts/redis.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP Warning:  PHP Startup: dbus: Unable to initialize module
Module compiled with build ID=API20190902,TS
PHP    compiled with build ID=API20190902,NTS
These options need to match
 in Unknown on line 0
PHP 7.4.33 (cli) (built: Aug 13 2024 20:38:45) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with the ionCube PHP Loader + ionCube24 v13.3.1, Copyright (c) 2002-2024, by ionCube Ltd.

The last 2 (repeating) errors in /usr/local/apnscp/storage/logs/starts.log:

EXCEPTION: Class 'Redis' not found 
[/usr/local/apnscp/lib/Cache/Mproxy.php:14]

         0B. include()
            [/usr/local/apnscp/lib/apnscpfunction.php:323]
         1B. apnscpFunctionInterceptor::autoload("Cache_Mproxy")
            [n/a]
         2B. spl_autoload_call("Cache_Mproxy")
            [/usr/local/apnscp/lib/config.php:282]
         3B. include("/usr/local/apnscp/lib/config.php")
            [/usr/local/apnscp/lib/constants.php:61]
         4B. {closure}()
            [/usr/local/apnscp/lib/constants.php:69]
         5B. include("/usr/local/apnscp/lib/constants.php")
            [/usr/local/apnscp/lib/apnscpcore.php:77]
         6B. include("/usr/local/apnscp/lib/apnscpcore.php")
            [/usr/local/apnscp/lib/lservice.php:25]
Failed loading /usr/local/apnscp/sys/php/20190902-zts/opcache.so:  /usr/local/apnscp/sys/php/20190902-zts/opcache.so: undefined symbol: zend_signal_globals_offs
et
PHP Warning:  PHP Startup: Unable to load dynamic library 'apcu.so' (tried: /usr/local/apnscp/sys/php/20190902-zts/apcu.so (/usr/local/apnscp/sys/php/20190902-z
ts/apcu.so: undefined symbol: executor_globals_offset), /usr/local/apnscp/sys/php/20190902-zts/apcu.so.so (/usr/local/apnscp/sys/php/20190902-zts/apcu.so.so: ca
nnot open shared object file: No such file or directory)) in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library 'igbinary.so' (tried: /usr/local/apnscp/sys/php/20190902-zts/igbinary.so (/usr/local/apnscp/sys/php/20
190902-zts/igbinary.so: undefined symbol: core_globals_offset), /usr/local/apnscp/sys/php/20190902-zts/igbinary.so.so (/usr/local/apnscp/sys/php/20190902-zts/ig
binary.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library 'redis.so' (tried: /usr/local/apnscp/sys/php/20190902-zts/redis.so (/usr/local/apnscp/sys/php/20190902
-zts/redis.so: undefined symbol: executor_globals_offset), /usr/local/apnscp/sys/php/20190902-zts/redis.so.so (/usr/local/apnscp/sys/php/20190902-zts/redis.so.s
o: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP Warning:  PHP Startup: dbus: Unable to initialize module
Module compiled with build ID=API20190902,TS
PHP    compiled with build ID=API20190902,NTS
These options need to match
 in Unknown on line 0

and…

[root@v1 apnscp]# git describe
v3.2.37
upcp -sbf apnscp/install-extensions

results in…

RUNNING HANDLER [common : Restart apnscp] **********************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unable to start service apnscp: Job for apnscp.service failed because the control process exited with error code.\nSee \"systemctl status apnscp.service\" and \"journalctl -xe\" for details.\n"}

NO MORE HOSTS LEFT *********************************************************************************************************************************************

PLAY RECAP *****************************************************************************************************************************************************
localhost                  : ok=69   changed=11   unreachable=0    failed=1    skipped=19   rescued=0    ignored=0   

which appears to be hung up on the old php version.