6.x blew out your network stack on RHEL8. You’ll need to manually revert back to 4.x either at boot or using grubby. Here’s what a quick Google search yielded.
Remember: experimental means just that, you are at risk of breaking something.
thanks 
i dont want to be expirimental only on dev system this is a production system. will check it out
now booted with kernal 8.9 4.18.0-513.5.1
still the same issus 
journalctl -n150 -u httpd assuming Apache is still giving you issues. Additional a scrub may be beneficial, upcp -sbf
journactl -n150 -u httpd
-bash: journactl: command not found
journalctl -n150 -u httpd
Mar 10 20:02:37 <servername>, systemd[1]: Starting The Apache HTTP Server...
Mar 10 20:02:37 <servername>, httpd.init[3525]: chown: invalid user: 'apache:apache'
Mar 10 20:02:37 <servername>, httpd.init[3533]: chown: invalid user: 'apache'
Mar 10 20:02:38 <servername>, systemd[1]: httpd.service: Got notification message from PID 3562, but reception only permitted for main PID which is currently not known
Mar 10 20:02:38 <servername>, httpd.init[3481]: Starting httpd:
Mar 10 20:02:38 <servername>, httpd.init[3562]: AH00543: httpd: bad user name apache
Mar 10 20:02:38 <servername>, httpd.init[3539]: [34B blob data]
Mar 10 20:02:38 <servername>, httpd.init[3633]: lsof: can't get UID for apache
Mar 10 20:02:38 <servername>, httpd.init[3633]: lsof 4.93.2
Mar 10 20:02:38 <servername>, httpd.init[3633]: latest revision: https://github.com/lsof-org/lsof
Mar 10 20:02:38 <servername>, httpd.init[3633]: latest FAQ: https://github.com/lsof-org/lsof/blob/master/00FAQ
Mar 10 20:02:38 <servername>, httpd.init[3633]: latest (non-formatted) man page: https://github.com/lsof-org/lsof/blob/master/Lsof.8
Mar 10 20:02:38 <servername>, httpd.init[3633]: usage: [-?abhKlnNoOPRtUvVX] [+|-c c] [+|-d s] [+D D] [+|-E] [+|-e s] [+|-f[gG]]
Mar 10 20:02:38 <servername>, httpd.init[3633]: [-F [f]] [-g [s]] [-i [i]] [+|-L [l]] [+m [m]] [+|-M] [-o [o]] [-p s]
Mar 10 20:02:38 <servername>, httpd.init[3633]: [+|-r [t]] [-s [p:s]] [-S [t]] [-T [t]] [-u s] [+|-w] [-x [fl]] [--] [names]
Mar 10 20:02:38 <servername>, httpd.init[3633]: Use the ``-h'' option to get more help information.
Mar 10 20:02:38 <servername>, httpd.init[3654]: lsof: can't get UID for apache
Mar 10 20:02:38 <servername>, httpd.init[3654]: lsof 4.93.2
Mar 10 20:02:38 <servername>, httpd.init[3654]: latest revision: https://github.com/lsof-org/lsof
Mar 10 20:02:38 <servername>, httpd.init[3654]: latest FAQ: https://github.com/lsof-org/lsof/blob/master/00FAQ
Mar 10 20:02:38 <servername>, httpd.init[3654]: latest (non-formatted) man page: https://github.com/lsof-org/lsof/blob/master/Lsof.8
Mar 10 20:02:38 <servername>, httpd.init[3654]: usage: [-?abhKlnNoOPRtUvVX] [+|-c c] [+|-d s] [+D D] [+|-E] [+|-e s] [+|-f[gG]]
Mar 10 20:02:38 <servername>, httpd.init[3654]: [-F [f]] [-g [s]] [-i [i]] [+|-L [l]] [+m [m]] [+|-M] [-o [o]] [-p s]
Mar 10 20:02:38 <servername>, httpd.init[3654]: [+|-r [t]] [-s [p:s]] [-S [t]] [-T [t]] [-u s] [+|-w] [-x [fl]] [--] [names]
Mar 10 20:02:38 <servername>, httpd.init[3654]: Use the ``-h'' option to get more help information.
Mar 10 20:02:38 <servername>, systemd[1]: httpd.service: Got notification message from PID 3682, but reception only permitted for main PID which is currently not known
Mar 10 20:02:38 <servername>, httpd.init[3682]: AH00543: httpd: bad user name apache
Mar 10 20:02:38 <servername>, httpd.init[3539]: [FAILED]
Mar 10 20:02:38 <servername>, systemd[1]: httpd.service: Can't open PID file /run/httpd.pid (yet?) after start: No such file or directory
Mar 10 20:02:38 <servername>, systemd[1]: httpd.service: Failed with result 'protocol'.
Mar 10 20:02:38 <servername>, systemd[1]: Failed to start The Apache HTTP Server.
says my hostname of the server
upcp -sbf
FAILED - RETRYING: CentOS 8+ package management (2 retries left).
FAILED - RETRYING: CentOS 8+ package management (1 retries left).
TASK [common/update-config : CentOS 8+ package management] **********************************************************************************************************************************
fatal: [localhost]: FAILED! => {"attempts": 2, "changed": true, "cmd": ["dnf", "remove", "-y", "--duplicates", "--setopt=protected_packages="], "delta": "0:00:01.210679", "end": "2024-03-10 20:20:02.315117", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2024-03-10 20:20:01.104438", "stderr": "Unable to detect release version (use '--releasever' to specify release version)\nErrors during downloading metadata for repository 'apnscp':\n - Status code: 404 for https://yum.apnscp.com/os/$releasever/x86_64/repodata/repomd.xml (IP: 45.32.193.57)\nError: Failed to download metadata for repo 'apnscp': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "stderr_lines": ["Unable to detect release version (use '--releasever' to specify release version)", "Errors during downloading metadata for repository 'apnscp':", " - Status code: 404 for https://yum.apnscp.com/os/$releasever/x86_64/repodata/repomd.xml (IP: 45.32.193.57)", "Error: Failed to download metadata for repo 'apnscp': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried"], "stdout": "ApisCP Packages for CentOS Linux $releasever - 250 B/s | 239 B 00:00 ", "stdout_lines": ["ApisCP Packages for CentOS Linux $releasever - 250 B/s | 239 B 00:00 "]}
PLAY RECAP **********************************************************************************************************************************************************************************
localhost : ok=34 changed=0 unreachable=0 failed=1 skipped=19 rescued=0 ignored=0
Either user “apache” doesn’t exist in /etc/passwd or NSS is busted.
upcp -sbf
Mar 10 20:02:37 <servername>, httpd.init[3525]: chown: invalid user: 'apache:apache'
Mar 10 20:02:37 <servername>, httpd.init[3533]: chown: invalid user: 'apache'
Mar 10 20:02:38 <servername>, systemd[1]: httpd.service: Got notification message from PID 3562, but reception only permitted for main PID which is currently not known
Mar 10 20:02:38 <servername>, httpd.init[3481]: Starting httpd:
Mar 10 20:02:38 <servername>, httpd.init[3562]: AH00543: httpd: bad user name apache
"Unable to detect release version (use '--releasever' to specify release version)"
yum reinstall -y alma-release (or rocky-release if on Rocky).
There’s a file, /etc/redhat-release, that should exist and is sourced by dnf to determine your OS release.
yum reinstall -y rocky-release
Unable to detect release version (use '--releasever' to specify release version)
ApisCP Packages for CentOS Linux $releasever - x86_64 248 B/s | 239 B 00:00
Errors during downloading metadata for repository 'apnscp':
- Status code: 404 for https://yum.apnscp.com/os/$releasever/x86_64/repodata/repomd.xml (IP: 45.32.193.57)
Error: Failed to download metadata for repo 'apnscp': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
Specify --releasever=8. Here’s a link that I used to help determine this advice, that may be of use to you in the future.
Thanks
yum reinstall -y rocky-release --releasever=8
timescale_timescaledb 419 B/s | 833 B 00:01
PostgreSQL common RPMs for RHEL / Rocky 8 - x86_64 873 B/s | 659 B 00:00
PostgreSQL common RPMs for RHEL / Rocky 8 - x86_64 0.0 B/s | 0 B 00:00
Errors during downloading metadata for repository 'pgdg-common':
- Curl error (37): Couldn't read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG]
Error: Failed to retrieve GPG key for repo 'pgdg-common': Curl error (37): Couldn't read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG]
Let’s take a step back.
/etc/pki is missing. /etc/redhat-release is missing. What did you do, rm -rf /etc/*? Your system may not be in a salvageable condition if this is the case. If you didn’t, then your node experienced massive filesystem corruption, which it may not be in a salvageable condition.
The only advice I can provide at this point is try to get dnf working again. Look for the files in /etc/yum/repos.d/ that contain the bad repo, then remove this repo configuration. upcp -sbf will reprovision this configuration once it can run otherwise it may be time to flatten and reinstall as the system is in an unusable state.
oef
i did not remove /etc/
will try to fix the repos
yum reinstall -y rocky-release --releasever=8
Last metadata expiration check: 0:00:23 ago on Sun 10 Mar 2024 08:58:14 PM CET.
Package rocky-release available, but not installed.
No match for argument: rocky-release
Error: No packages marked for reinstall.
upcp -sbf
...
upcp -sbf
id: ‘apnscp’: no such user
id: ‘apnscp’: no such user
chown: invalid group: ‘root:apnscp’
setpriv: failed to parse reuid: '--regid'
setpriv: failed to parse reuid: '--regid'
setpriv: failed to parse reuid: '--regid'
...
chown: invalid user: ‘apnscp:apnscp’
WARNING: Cache_Mproxy::_connect(): cache: cannot access server
(RedisException) EXCEPTION: Redis server went away
[/usr/local/apnscp/lib/Cache/Mproxy.php:108]
...
TASK [filesystem/make-mounts : Verify quota package installed] ******************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: Curl error (37): Couldn't read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-apnscp [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-apnscp]
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/dnf/util.py\", line 115, in _urlopen\n repo._repo.downloadUrl(url, fo.fileno())\n File \"/usr/lib64/python3.6/site-packages/libdnf/repo.py\", line 499, in downloadUrl\n return _repo.Repo_downloadUrl(self, url, fd)\nRuntimeError: Curl error (37): Couldn't read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-apnscp [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-apnscp]\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_dnf_payload_jkd9c7vg/ansible_dnf_payload.zip/ansible/modules/packaging/os/dnf.py\", line 1370, in <module>\n File \"/tmp/ansible_dnf_payload_jkd9c7vg/ansible_dnf_payload.zip/ansible/modules/packaging/os/dnf.py\", line 1359, in main\n File \"/tmp/ansible_dnf_payload_jkd9c7vg/ansible_dnf_payload.zip/ansible/modules/packaging/os/dnf.py\", line 1338, in run\n File \"/tmp/ansible_dnf_payload_jkd9c7vg/ansible_dnf_payload.zip/ansible/modules/packaging/os/dnf.py\", line 1242, in ensure\n File \"/usr/lib/python3.6/site-packages/dnf/base.py\", line 2494, in _get_key_for_package\n keys = dnf.crypto.retrieve(keyurl, repo)\n File \"/usr/lib/python3.6/site-packages/dnf/crypto.py\", line 185, in retrieve\n with dnf.util._urlopen(keyurl, repo=repo) as handle:\n File \"/usr/lib/python3.6/site-packages/dnf/util.py\", line 119, in _urlopen\n raise IOError(str(e))\nOSError: Curl error (37): Couldn't read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-apnscp [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-apnscp]\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
PLAY RECAP **********************************************************************************************************************************************************************************
localhost : ok=56 changed=0 unreachable=0 failed=1 skipped=28 rescued=0 ignored=0
TASK [packages/install : Link PGDG-13 to PGDG] **********************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "src file does not exist, use \"force=yes\" if you really want to create the link: /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-13", "path": "/etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG", "src": "/etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-13"}
PLAY RECAP **********************************************************************************************************************************************************************************
localhost : ok=126 changed=0 unreachable=0 failed=1 skipped=59 rescued=0 ignored=0
TASK [packages/install : Install packages] ************************************************************************************************************************
fatal: [localhost]: FAILED! => {"attempts": 2, "changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem 1: package MariaDB-server-10.5.24-1.el8.x86_64 from mariadb requires perl(DBI), but none of the providers can be installed\n - cannot install the best candidate for the job\n - package perl-DBI-1.641-1.el8.x86_64 from baseos is filtered out by modular filtering\n Problem 2: package MariaDB-client-10.5.24-1.el8.x86_64 from mariadb requires perl(DBI), but none of the providers can be installed\n - cannot install the best candidate for the job\n - package perl-DBI-1.641-1.el8.x86_64 from baseos is filtered out by modular filtering", "rc": 1, "results": []}
PLAY RECAP ********************************************************************************************************************************************************
localhost : ok=135 changed=4 unreachable=0 failed=1 skipped=67 rescued=0 ignored=0
You’ll need to get your system back into a usable state. This was caused by filesystem corruption as your RPM database (/var/lib/rpm/Packages) is corrupted, GPG keys in /etc/pki are missing, CentOS/Redhat release file missing thus affecting $releasever macro expansion, and NSS is blown out the water.
getent passwd apache would query /etc/nsswitch.conf for the “passwd” field. In a normal system it looks like passwd: sss files systemd apnscpvwh. It in turn looks at /etc/passwd for the user apache.
This field isn’t missing unless /etc/passwd is corrupted. If /etc/passwd is corrupted, then your hard drive is on its way to binary heaven.
Can u please help me out with remote support?
getent passwd apache
gives me
apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin
You may purchase a Silver support plan under my.apiscp.com for hands time on your server.
In the meanwhile, I’d advise running a SMART check on the drive. If it’s backed by a thinly-provisioned LVM pool, be careful as exhausting metadata can lead to corruption as well.
thanks for al your help matt!
i have fdisk the servers hdds no errors so far… will continu scanning.
installed everything by hand (pain in the assss) on a new server.
looks like everything is working for now.
thanks again!