a lot of problemos

msaladna · March 10, 2024, 7:21pm

Either user “apache” doesn’t exist in /etc/passwd or NSS is busted.

upcp -sbf

Mar 10 20:02:37 <servername>, httpd.init[3525]: chown: invalid user: 'apache:apache'
Mar 10 20:02:37 <servername>, httpd.init[3533]: chown: invalid user: 'apache'
Mar 10 20:02:38 <servername>, systemd[1]: httpd.service: Got notification message from PID 3562, but reception only permitted for main PID which is currently not known
Mar 10 20:02:38 <servername>, httpd.init[3481]: Starting httpd:
Mar 10 20:02:38 <servername>, httpd.init[3562]: AH00543: httpd: bad user name apache

"Unable to detect release version (use '--releasever' to specify release version)"

yum reinstall -y alma-release (or rocky-release if on Rocky).

There’s a file, /etc/redhat-release, that should exist and is sourced by dnf to determine your OS release.

greensmart01 · March 10, 2024, 7:26pm

yum reinstall -y rocky-release
Unable to detect release version (use '--releasever' to specify release version)
ApisCP Packages for CentOS Linux $releasever - x86_64                                                                                                        248  B/s | 239  B     00:00
Errors during downloading metadata for repository 'apnscp':
  - Status code: 404 for https://yum.apnscp.com/os/$releasever/x86_64/repodata/repomd.xml (IP: 45.32.193.57)
Error: Failed to download metadata for repo 'apnscp': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

msaladna · March 10, 2024, 7:28pm

Specify --releasever=8. Here’s a link that I used to help determine this advice, that may be of use to you in the future.

greensmart01 · March 10, 2024, 7:44pm

Thanks

 yum reinstall -y rocky-release --releasever=8
timescale_timescaledb                                                                                                                                        419  B/s | 833  B     00:01
PostgreSQL common RPMs for RHEL / Rocky 8 - x86_64                                                                                                           873  B/s | 659  B     00:00
PostgreSQL common RPMs for RHEL / Rocky 8 - x86_64                                                                                                           0.0  B/s |   0  B     00:00
Errors during downloading metadata for repository 'pgdg-common':
  - Curl error (37): Couldn't read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG]
Error: Failed to retrieve GPG key for repo 'pgdg-common': Curl error (37): Couldn't read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG]

msaladna · March 10, 2024, 7:48pm

Let’s take a step back.

/etc/pki is missing. /etc/redhat-release is missing. What did you do, rm -rf /etc/*? Your system may not be in a salvageable condition if this is the case. If you didn’t, then your node experienced massive filesystem corruption, which it may not be in a salvageable condition.

The only advice I can provide at this point is try to get dnf working again. Look for the files in /etc/yum/repos.d/ that contain the bad repo, then remove this repo configuration. upcp -sbf will reprovision this configuration once it can run otherwise it may be time to flatten and reinstall as the system is in an unusable state.

greensmart01 · March 10, 2024, 7:52pm

oef

i did not remove /etc/

will try to fix the repos

greensmart01 · March 10, 2024, 7:59pm

 yum reinstall -y rocky-release --releasever=8
Last metadata expiration check: 0:00:23 ago on Sun 10 Mar 2024 08:58:14 PM CET.
Package rocky-release available, but not installed.
No match for argument: rocky-release
Error: No packages marked for reinstall.

greensmart01 · March 10, 2024, 8:22pm

upcp -sbf

...

 upcp -sbf
id: ‘apnscp’: no such user
id: ‘apnscp’: no such user
chown: invalid group: ‘root:apnscp’
setpriv: failed to parse reuid: '--regid'
setpriv: failed to parse reuid: '--regid'
setpriv: failed to parse reuid: '--regid'

...

chown: invalid user: ‘apnscp:apnscp’
WARNING: Cache_Mproxy::_connect(): cache: cannot access server
(RedisException) EXCEPTION: Redis server went away
[/usr/local/apnscp/lib/Cache/Mproxy.php:108]


...



TASK [filesystem/make-mounts : Verify quota package installed] ******************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: Curl error (37): Couldn't read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-apnscp [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-apnscp]
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/usr/lib/python3.6/site-packages/dnf/util.py\", line 115, in _urlopen\n    repo._repo.downloadUrl(url, fo.fileno())\n  File \"/usr/lib64/python3.6/site-packages/libdnf/repo.py\", line 499, in downloadUrl\n    return _repo.Repo_downloadUrl(self, url, fd)\nRuntimeError: Curl error (37): Couldn't read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-apnscp [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-apnscp]\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"<stdin>\", line 102, in <module>\n  File \"<stdin>\", line 94, in _ansiballz_main\n  File \"<stdin>\", line 40, in invoke_module\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_dnf_payload_jkd9c7vg/ansible_dnf_payload.zip/ansible/modules/packaging/os/dnf.py\", line 1370, in <module>\n  File \"/tmp/ansible_dnf_payload_jkd9c7vg/ansible_dnf_payload.zip/ansible/modules/packaging/os/dnf.py\", line 1359, in main\n  File \"/tmp/ansible_dnf_payload_jkd9c7vg/ansible_dnf_payload.zip/ansible/modules/packaging/os/dnf.py\", line 1338, in run\n  File \"/tmp/ansible_dnf_payload_jkd9c7vg/ansible_dnf_payload.zip/ansible/modules/packaging/os/dnf.py\", line 1242, in ensure\n  File \"/usr/lib/python3.6/site-packages/dnf/base.py\", line 2494, in _get_key_for_package\n    keys = dnf.crypto.retrieve(keyurl, repo)\n  File \"/usr/lib/python3.6/site-packages/dnf/crypto.py\", line 185, in retrieve\n    with dnf.util._urlopen(keyurl, repo=repo) as handle:\n  File \"/usr/lib/python3.6/site-packages/dnf/util.py\", line 119, in _urlopen\n    raise IOError(str(e))\nOSError: Curl error (37): Couldn't read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-apnscp [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-apnscp]\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

PLAY RECAP **********************************************************************************************************************************************************************************
localhost                  : ok=56   changed=0    unreachable=0    failed=1    skipped=28   rescued=0    ignored=0

greensmart01 · March 10, 2024, 8:49pm

TASK [packages/install : Link PGDG-13 to PGDG] **********************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "src file does not exist, use \"force=yes\" if you really want to create the link: /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-13", "path": "/etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG", "src": "/etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-13"}

PLAY RECAP **********************************************************************************************************************************************************************************
localhost                  : ok=126  changed=0    unreachable=0    failed=1    skipped=59   rescued=0    ignored=0

greensmart01 · March 10, 2024, 9:45pm

TASK [packages/install : Install packages] ************************************************************************************************************************
fatal: [localhost]: FAILED! => {"attempts": 2, "changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem 1: package MariaDB-server-10.5.24-1.el8.x86_64 from mariadb requires perl(DBI), but none of the providers can be installed\n  - cannot install the best candidate for the job\n  - package perl-DBI-1.641-1.el8.x86_64 from baseos is filtered out by modular filtering\n Problem 2: package MariaDB-client-10.5.24-1.el8.x86_64 from mariadb requires perl(DBI), but none of the providers can be installed\n  - cannot install the best candidate for the job\n  - package perl-DBI-1.641-1.el8.x86_64 from baseos is filtered out by modular filtering", "rc": 1, "results": []}

PLAY RECAP ********************************************************************************************************************************************************
localhost                  : ok=135  changed=4    unreachable=0    failed=1    skipped=67   rescued=0    ignored=0

msaladna · March 11, 2024, 2:55pm

You’ll need to get your system back into a usable state. This was caused by filesystem corruption as your RPM database (/var/lib/rpm/Packages) is corrupted, GPG keys in /etc/pki are missing, CentOS/Redhat release file missing thus affecting $releasever macro expansion, and NSS is blown out the water.

getent passwd apache would query /etc/nsswitch.conf for the “passwd” field. In a normal system it looks like passwd: sss files systemd apnscpvwh. It in turn looks at /etc/passwd for the user apache.

This field isn’t missing unless /etc/passwd is corrupted. If /etc/passwd is corrupted, then your hard drive is on its way to binary heaven.

greensmart01 · March 11, 2024, 4:17pm

Can u please help me out with remote support?

greensmart01 · March 11, 2024, 6:06pm

getent passwd apache

gives me

apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin

msaladna · March 12, 2024, 1:13am

You may purchase a Silver support plan under my.apiscp.com for hands time on your server.

In the meanwhile, I’d advise running a SMART check on the drive. If it’s backed by a thinly-provisioned LVM pool, be careful as exhausting metadata can lead to corruption as well.

greensmart01 · March 13, 2024, 6:59pm

thanks for al your help matt!

i have fdisk the servers hdds no errors so far… will continu scanning.

installed everything by hand (pain in the assss) on a new server.
looks like everything is working for now.
thanks again!