Repairing extremely out of date server

I have a personal VPS running Centos7 and ApisCP for some personal projects and email and i thought it had been auto updating. I had to go in yesterday to update some mail settings and i saw that it was super out of date. I read through a couple forum threads to try to fix it but I was unable to update the server (yum update) due to the OS being EOL. At this point i think it’ll be quickest to just wipe the box and start fresh. To accomplish that I have two questions:

  1. Is there a preferred OS for a fresh install? It looks like my options are Rocky or AlmaLinux?
  2. Is there a list of files I need to create a backup so I can restore my settings post wipe?

Thanks!

You may either perform an in-place upgrade using LEAPP or migrate all sites to a new platform using the migration tool. Unless you’re emotionally attached to that machine, migrating sites to a new platform is the best option as it gives you an opportunity to troubleshoot any sites individually before migrating to a new environment.

I’d recommend Rocky 8 at this point.

AlmaLinux and Rocky Linux set out to be bug-for-bug rebuilds of RHEL8. Alma walked back on this pledge, Rocky has not. I’ve seen two separate users running into system hangs on fresh install with Alma that could not be reproduced with Rocky. ABI compatibility, what AlmaLinux is now, is a much weaker standard than 1:1 rebuilds, what Rocky still is. With the complexity and criticality of both kernel and systemd - the more engineering resources on code the better it is for stability.

I tried LEAPP but it failed - most likely due to the fact that i can’t run a yum update successfully. Does the migration tool require upcp to be working? Because at this point it’s not. I think i might be at the manual backup and wipe stage.

So long as the panel works, you can migrate off it.

On LEAPP, looks like CentOS pulled the plug on the repo entirely. Rocky has provided a workaround for this if you’d like to do in-place. I’d still recommend a server migration.

ImportDomain is not working either.

ImportDomain
apnscp_php: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such file or directory

Are there steps anywhere for a manual process?

You’d want the migration tool, not import tool which works with cPanel backups at this time. Only cpcmd is required to work for this tool, which looks to be bad from a missing readline library?

upcp -sbf apnscp/build-php apnscp/install-extensions

Let’s see if platform PHP rebuilds OK.

No luck there either. I’ve got a mess of mismatched dependancies

What errors do you receive when attempting to run the command? These guide me in helping you find a solution otherwise it’s impossible to know with certainty.

if i try to run apnscp/install-extensions i get:
TASK [php/install-pecl-module : Update channel protocols] ************************************************************************************************************************************************************************************************ fatal: [localhost]: FAILED! => {"attempts": 3, "changed": true, "cmd": ["/usr/bin/pecl", "channel-update", "pecl.php.net"], "delta": "0:00:00.008913", "end": "2024-08-22 21:59:51.065534", "msg": "non-zero return code", "rc": 127, "start": "2024-08-22 21:59:51.056621", "stderr": "/usr/local/apnscp/bin/php-bins/apnscp_php: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such file or directory", "stderr_lines": ["/usr/local/apnscp/bin/php-bins/apnscp_php: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such file or directory"], "stdout": "", "stdout_lines": []}

With apnscp/build-php first i get the attached error.
php-step-error.txt.zip (6.7 KB)

What versions of python and ansible should I be running?

Looks like you’re running an ancient release. Oniguruma was added June 2020 as a build dependency

There are two approaches here:

First just linking libreadline.so.5 to 6,

ln -s /usr/lib64/libreadline.so.{6,5}

See if apnscp_php -v works then. If so, and it reports PHP 8.1.x, then you’re good to go. If it reports 7.4.x, then there may be API changes preventing a server-to-server migration. Run through the migration process; report any errors encountered as this is terra incognita and I’m not much for reading minds.

Alternatively, to rebuild panel PHP, the first step is to correct the yum repos in /etc/yum.repos.d, see this article. Once that’s done verify it works with yum install -y oniguruma-devel dbus-devel, then proceed with upcp -sbf apnscp/build-php apnscp/install-extensions

Note this order is important and must be done in serial.

I went through and fixed all the symlink errors.

apnscp_php -v is returning: apnscp_php: symbol lookup error: apnscp_php: undefined symbol: gdbm_errno

build-php is returning a bunch of missing imports. i can’t figure out what’s missing there. here’s a sample:

/usr/local/apnscp/build/php/php-7.4.33/ext/date/php_date.c:4231: error: undefined reference to 'executor_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/date/php_date.c:2089: error: undefined reference to 'executor_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/date/php_date.c:5439: error: undefined reference to 'executor_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/date/php_date.c:5439: error: undefined reference to 'executor_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/libxml/libxml.c:359: error: undefined reference to 'file_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/libxml/libxml.c:762: error: undefined reference to 'core_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/openssl/xp_ssl.c:2451: error: undefined reference to 'file_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/openssl/xp_ssl.c:2674: error: undefined reference to 'file_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/openssl/xp_ssl.c:2674: error: undefined reference to 'file_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/pcre/php_pcre.c:602: error: undefined reference to 'basic_globals'
/usr/local/apnscp/build/php/php-7.4.33/Zend/zend_hash.h:813: error: undefined reference to 'basic_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/pcre/php_pcre.c:775: error: undefined reference to 'basic_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/sqlite3/sqlite3.c:164: error: undefined reference to 'core_globals'
/usr/local/apnscp/build/php/php-7.4.33/Zend/zend_types.h:441: error: undefined reference to 'core_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/zlib/zlib.c:240: error: undefined reference to 'sapi_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/curl/interface.c:2579: error: undefined reference to 'core_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/ffi/ffi.c:2849: error: undefined reference to 'compiler_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/ffi/ffi.c:2849: error: undefined reference to 'compiler_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/ffi/ffi.c:2849: error: undefined reference to 'compiler_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/ffi/ffi.c:2849: error: undefined reference to 'compiler_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/filter/sanitizing_filters.c:266: error: undefined reference to 'sapi_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/iconv/iconv.c:390: error: undefined reference to 'sapi_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/mbstring/mbstring.c:2244: error: undefined reference to 'sapi_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/phar/phar_object.c:4704: error: undefined reference to 'basic_globals'
/usr/local/apnscp/build/php/php-7.4.33/ext/soap/php_sdl.c:173: error: undefined reference to 'soap_globals_id'
/usr/local/apnscp/build/php/php-7.4.33/ext/soap/php_sdl.c:173: error: undefined reference to 'soap_globals_id'
/usr/local/apnscp/build/php/php-7.4.33/ext/soap/php_sdl.c:173: error: undefined reference to 'soap_globals_id'
/usr/local/apnscp/build/php/php-7.4.33/ext/soap/php_sdl.c:173: error: undefined reference to 'soap_globals_id'
/usr/local/apnscp/build/php/php-7.4.33/TSRM/TSRM.c:527: error: undefined reference to 'pthread_getspecific'
/usr/local/apnscp/build/php/php-7.4.33/TSRM/TSRM.c:533: error: undefined reference to 'pthread_setspecific'
/usr/local/apnscp/build/php/php-7.4.33/TSRM/TSRM.c:238: error: undefined reference to 'pthread_setspecific'
/usr/local/apnscp/build/php/php-7.4.33/TSRM/TSRM.c:239: error: undefined reference to 'pthread_key_delete'
/usr/local/apnscp/build/php/php-7.4.33/TSRM/TSRM.c:406: error: undefined reference to 'pthread_setspecific'
/usr/local/apnscp/build/php/php-7.4.33/TSRM/TSRM.c:406: error: undefined reference to 'pthread_setspecific'
/usr/local/apnscp/build/php/php-7.4.33/TSRM/TSRM.c:549: error: undefined reference to 'pthread_getspecific'
/usr/local/apnscp/build/php/php-7.4.33/TSRM/TSRM.c:448: error: undefined reference to 'pthread_getspecific'
/usr/local/apnscp/build/php/php-7.4.33/TSRM/TSRM.c:142: error: undefined reference to 'pthread_key_create'
/usr/local/apnscp/build/php/php-7.4.33/TSRM/TSRM.c:760: error: undefined reference to 'pthread_sigmask'
/usr/local/apnscp/build/php/php-7.4.33/TSRM/TSRM.c:842: error: undefined reference to 'pthread_getspecific'

Is there a specific version of python and ansible i should be using?

rm -rf /usr/local/apnscp/build/php/php-7.4.33/

Then re-run upcp -sbf apnscp/build-php apnscp/install-extensions. Later panel iterations cleanup build environment, running something 4 years old… is bound to have its moments.

Ansible and Python are locked to OS. It’s irrelevant in this context.

Getting closer. The build finished this time but I’m getting two errors on startup:

PHP Warning:  PHP Startup: Unable to load dynamic library 'event.so' (tried: /usr/local/apnscp/sys/php/20190902/event.so (libevent_openssl-2.0.so.5: cannot open shared object file: No such file or directory), /usr/local/apnscp/sys/php/20190902/event.so.so (/usr/local/apnscp/sys/php/20190902/event.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP Parse error:  syntax error, unexpected '|', expecting '{' in /usr/local/apnscp/lib/helpers.php on line 181

Thank you for all your help so far by the way!

This smells similar to another situation that left off at 3.2.3x.

Code is referencing PHP 8.1, right now you’re on 7.4. Lockstep migrations were introduced ~18 months ago. Let’s approach this as if you’re at the crescent end of PHP 7.4, then move to 8.1:

First perform a full platform scrub:

upcp -sbf

Then jump back to 3.2.37, which was the last release before 8.1:

git checkout v3.2.37
chown -R apnscp /usr/local/apnscp
systemctl restart apiscp
./artisan migrate --force

If all goes accordingly, then you should be here for the last remnant to bring it up to date.

no luck on the migrations. first one failed.

TASK [php/install-pecl-module : Build and install imagick library.] ************
fatal: [localhost]: FAILED! => changed=true 
  cmd:
  - /usr/local/apnscp/resources/playbooks/roles/php/install-pecl-module/files/installPecl.sh
  - imagick
  - ''
  delta: '0:00:02.675566'
  end: '2024-08-23 15:27:48.862378'
  msg: non-zero return code
  rc: 1
  start: '2024-08-23 15:27:46.186812'
  stderr: |-
    configure: WARNING: you should use --build, --host, --target
    configure: error: not found. Please provide a path to MagickWand-config or Wand-config program.
  stderr_lines: <omitted>
  stdout: |-
    /usr/local/apnscp/storage/tmp/extension.EuquTeT /usr/local/apnscp/resources/playbooks
    downloading imagick-3.7.0.tgz ...
    Starting to download imagick-3.7.0.tgz (360,138 bytes)
    .........................................................................done: 360,138 bytes
    File /usr/local/apnscp/storage/tmp/extension.EuquTeT/imagick-3.7.0.tgz downloaded
    /usr/local/apnscp/storage/tmp/extension.EuquTeT/imagick-3.7.0 /usr/local/apnscp/storage/tmp/extension.EuquTeT /usr/local/apnscp/resources/playbooks
    Configuring for:
    PHP Api Version:         20190902
    Zend Module Api No:      20190902
    Zend Extension Api No:   320190902
    checking for grep that handles long lines and -e... /bin/grep
    checking for egrep... /bin/grep -E
    checking for a sed that does not truncate output... /bin/sed
    checking for pkg-config... /bin/pkg-config
    checking pkg-config is at least version 0.9.0... yes
    checking for cc... cc
    checking whether the C compiler works... yes
    checking for C compiler default output file name... a.out
    checking for suffix of executables...
    checking whether we are cross compiling... no
    checking for suffix of object files... o
    checking whether we are using the GNU C compiler... yes
    checking whether cc accepts -g... yes
    checking for cc option to accept ISO C89... none needed
    checking how to run the C preprocessor... cc -E
    checking for icc... no
    checking for suncc... no
    checking for system library directory... lib
    checking if compiler supports -R... no
    checking if compiler supports -Wl,-rpath,... yes
    checking build system type... x86_64-pc-linux-gnu
    checking host system type... x86_64-pc-linux-gnu
    checking target system type... x86_64-pc-linux-gnu
    checking for PHP prefix... /usr
    checking for PHP includes... -I/usr/include/php -I/usr/include/php/main -I/usr/include/php/TSRM -I/usr/include/php/Zend -I/usr/include/php/ext -I/usr/include/php/ext/date/lib
    checking for PHP extension directory... /usr/lib64/php/modules
    checking for PHP installed headers prefix... /usr/include/php
    checking if debug is enabled... no
    checking if zts is enabled... no
    checking for gawk... gawk
    checking whether to enable the imagick extension... yes, shared
    checking for pkg-config... /bin/pkg-config
    checking ImageMagick MagickWand API configuration program... checking Testing /usr/local/bin/MagickWand-config... Doesn't exist
    checking Testing /usr/bin/MagickWand-config... Doesn't exist
    checking Testing /usr/sbin/bin/MagickWand-config... Doesn't exist
    checking Testing /opt/bin/MagickWand-config... Doesn't exist
    checking Testing /opt/local/bin/MagickWand-config... Doesn't exist
    checking Testing /opt/homebrew/bin/MagickWand-config... Doesn't exist
  stdout_lines: <omitted>

PLAY RECAP *********************************************************************
localhost                  : ok=18   changed=0    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0  

yum install -y ImageMagick-devel, then run again

Slowly getting through the migrations. I’m stuck on this error though:

TASK [apache/modsecurity : Install mod_security package] ***********************
fatal: [localhost]: FAILED! => changed=false 
  failures:
  - No package mod_security-clamav-connector available.
  msg: Failed to install some of the specified packages
  rc: 1
  results: []

That’s part of the apnscp repo, which you might’ve disabled as well?

yum reinstall -y http://yum.apnscp.com/apnscp-release-latest-7.noarch.rpm

Works fine for me when “apnscp” repo enabled:

# yum reinstall -y mod_security-clamav-connector

4 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package mod_security-clamav-connector.noarch 0:2.0-2.apnscp will be reinstalled
--> Finished Dependency Resolution

Since the server is getting retired, disable ClamAV:

cpcmd scope:set cp.bootstrapper clamav_enabled False

Then run upcp -sbf again.