Looks like my setup is broken again 
?
websites are running. -i can access them.
server.domain/mail is running - i cannot access with login credentionals.
i have can shh into the server.
mails in /root/new are saying;
Does not exist apnscp
please help me
-- Unit apnscp.service has begun starting up.
May 02 00:43:51 <servername> systemd[1]: apnscp.service: Supervising process 19639 which is not our child. We'll >
May 02 00:43:51 <servername> systemd[1]: apnscp.service: Main process exited, code=exited, status=255/n/a
May 02 00:43:51 <servername> httpd[19641]: httpd: Syntax error on line 82 of /usr/local/apnscp/config/httpd.conf:>
May 02 00:43:51 <servername> systemd[1]: apnscp.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The unit apnscp.service has entered the 'failed' state with result 'exit-code'.
May 02 00:43:51 <servername> systemd[1]: Failed to start ApisCP backend service.
-- Subject: Unit apnscp.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit apnscp.service has failed.
--
-- The result is failed.
Hard to say. Did you run over your storage again? rm -rf /usr/local/apnscp/storage/cache/* then restart ApisCP, systemctl restart apiscp.
Thanks for the quick reply Matt 
rm -rf /usr/local/apnscp/storage/cache/*
systemctl restart apiscp
Job for apnscp.service failed because the control process exited with error code .
See "systemctl status apnscp.service" and "journalctl -xe" for details.
journalctl -xe
May 02 00:58:20 <servername> auth[18016]: auth worker: PASSV: pam_unix(imap:auth): authentication failure; logname= uid=0 e>
May 02 00:58:24 <servername> auth[18097]: auth worker: PASSV: pam_listfile(imap:auth): Couldn't open /etc/imap.pamlist
May 02 00:58:24 <servername> auth[18097]: auth worker: PASSV: pam_unix(imap:auth): check pass; user unknown
May 02 00:58:24 <servername> auth[18097]: auth worker: PASSV: pam_unix(imap:auth): authentication failure; logname= uid=0 e>
May 02 00:58:29 <servername> dovecot[3749]: imap-login: Disconnected: Aborted login by logging out (auth failed, 2 attempts>
May 02 00:58:29 <servername> auth[18136]: auth worker: PASSV: pam_listfile(imap:auth): Couldn't open /etc/imap.pamlist
May 02 00:58:29 <servername> auth[18136]: auth worker: PASSV: pam_unix(imap:auth): check pass; user unknown
May 02 00:58:29 <servername> auth[18136]: auth worker: PASSV: pam_unix(imap:auth): authentication failure; logname= uid=0 e>
May 02 00:58:33 <servername> auth[18184]: auth worker: PASSV: pam_listfile(imap:auth): Couldn't open /etc/imap.pamlist
May 02 00:58:33 <servername> auth[18184]: auth worker: PASSV: pam_unix(imap:auth): check pass; user unknown
May 02 00:58:33 <servername> auth[18184]: auth worker: PASSV: pam_unix(imap:auth): authentication failure; logname= uid=0 e>
May 02 00:58:37 <servername> dovecot[3749]: imap-login: Disconnected: Aborted login by logging out (auth failed, 2 attempts>
May 02 00:58:37 <servername> auth[18226]: auth worker: PASSV: pam_listfile(imap:auth): Couldn't open /etc/imap.pamlist
May 02 00:58:37 <servername> auth[18226]: auth worker: PASSV: pam_unix(imap:auth): check pass; user unknown
cd /usr/local/apnscp/bin
./apnscpd -f restart
./apnscpd -f restart
PHP Fatal error: Uncaught Error: Class "Error_Reporter" not found in /usr/local/apnscp/lib/log_wrapper.php:14
Stack trace:
#0 /usr/local/apnscp/lib/apnscpcore.php(35): include()
#1 /usr/local/apnscp/lib/lservice.php(25): include('/usr/local/apns...')
#2 {main}
thrown in /usr/local/apnscp/lib/log_wrapper.php on line 14
Fatal error: Uncaught Error: Class "Error_Reporter" not found in /usr/local/apnscp/lib/log_wrapper.php:14
Stack trace:
#0 /usr/local/apnscp/lib/apnscpcore.php(35): include()
#1 /usr/local/apnscp/lib/lservice.php(25): include('/usr/local/apns...')
#2 {main}
thrown in /usr/local/apnscp/lib/log_wrapper.php on line 14
upcp --reset
Then give it another go.
upcp --reset
fatal: not a git repository (or any of the parent directories): .git
Reasserting permissions
storage/cache/e5f3d606bb6f9d11efd827797f51f0ee
storage/cache/e5f3d606bb6f9d11efd827797f51f0ee/usr
storage/cache/e5f3d606bb6f9d11efd827797f51f0ee/usr/local
storage/cache/e5f3d606bb6f9d11efd827797f51f0ee/usr/local/apnscp
storage/cache/e5f3d606bb6f9d11efd827797f51f0ee/usr/local/apnscp/bin
storage/cache/e5f3d606bb6f9d11efd827797f51f0ee/usr/local/apnscp/bin/apnscpd.bin
storage/cache/e5f3d606bb6f9d11efd827797f51f0ee/usr/local/apnscp/lib
storage/cache/e5f3d606bb6f9d11efd827797f51f0ee/usr/local/apnscp/lib/Opcenter
storage/cache/e5f3d606bb6f9d11efd827797f51f0ee/usr/local/apnscp/lib/Opcenter/License.php.bin
storage/cache/e5f3d606bb6f9d11efd827797f51f0ee/usr/local/apnscp/lib/Opcenter/Apnscp.php.bin
storage/cache/e5f3d606bb6f9d11efd827797f51f0ee/usr/local/apnscp/lib/Opcenter/System
storage/cache/e5f3d606bb6f9d11efd827797f51f0ee/usr/local/apnscp/lib/Opcenter/System/GenericSystemdService.php.bin
storage/cache/e5f3d606bb6f9d11efd827797f51f0ee/usr/local/apnscp/lib/apnscpfunction.php.bin
storage/tmp/agent-root
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Reasserting permissions
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
PHP Fatal error: Uncaught Error: Class "Error_Reporter" not found in /usr/local/apnscp/lib/log_wrapper.php:14
Stack trace:
#0 /usr/local/apnscp/lib/apnscpcore.php(35): include()
#1 /usr/local/apnscp/artisan(19): include('/usr/local/apns...')
#2 {main}
thrown in /usr/local/apnscp/lib/log_wrapper.php on line 14
You have new mail in /root/Mail
Yep, at some point you hit 100% storage.
cd /usr/local/apnscp
rm -rf .git/{objects,refs}
git clone https://gitlab.com/apisnetworks/apnscp.git ../apnscp-2
cp -dpR ../apnscp-2/.git/{objects,refs} .git/
git fetch origin
upcp --reset
cd /usr/local/apnscp
You have new mail in /root/Mail
rm -rf .git/{objects,refs}
git clone https://gitlab.com/apisnetworks/apnscp.git ../apnscp-2
Cloning into '../apnscp-2'...
remote: Enumerating objects: 66691, done.
remote: Counting objects: 100% (2819/2819), done.
remote: Compressing objects: 100% (1125/1125), done.
remote: Total 66691 (delta 1644), reused 2707 (delta 1586), pack-reused 63872 (from 1)
Receiving objects: 100% (66691/66691), 60.99 MiB | 61.96 MiB/s, done.
Resolving deltas: 100% (47317/47317), done.
cp -dpR ../apnscp-2/.git/{objects,refs} .git/
git fetch origin
fatal: not a git repository (or any of the parent directories): .git
upcp --reset
fatal: not a git repository (or any of the parent directories): .git
Reasserting permissions
.git/objects
.git/objects/pack
.git/objects/pack/pack-772e37d34805d3ed99728898fecdb245d13221ae.pack
.git/objects/pack/pack-772e37d34805d3ed99728898fecdb245d13221ae.idx
.git/objects/info
.git/refs
.git/refs/heads
.git/refs/heads/master
.git/refs/tags
.git/refs/remotes
.git/refs/remotes/origin
.git/refs/remotes/origin/HEAD
storage/tmp/agent-root
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Reasserting permissions
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
PHP Fatal error: Uncaught Error: Class "Error_Reporter" not found in /usr/local/apnscp/lib/log_wrapper.php:14
Stack trace:
#0 /usr/local/apnscp/lib/apnscpcore.php(35): include()
#1 /usr/local/apnscp/artisan(19): include('/usr/local/apns...')
#2 {main}
thrown in /usr/local/apnscp/lib/log_wrapper.php on line 14
Copy it all then, .git is toast.
cd /usr/local/apnscp
rm -rf .git
git clone https://gitlab.com/apisnetworks/apnscp.git ../apnscp-2
mv ../apnscp-2/.git ./
git fetch origin
upcp --reset
cd /usr/local/apnscp
rm -rf .git
git clone https://gitlab.com/apisnetworks/apnscp.git ../apnscp-2
fatal: destination path '../apnscp-2' already exists and is not an empty directory.
mv ../apnscp-2/.git ./
git fetch origin
upcp --reset
error: cannot open '.git/FETCH_HEAD': Permission denied
fatal: Unable to create '/usr/local/apnscp/.git/index.lock': Permission denied
Reasserting permissions
storage/tmp/agent-root
.git
.git/branches
.git/hooks
.git/hooks/applypatch-msg.sample
.git/hooks/commit-msg.sample
.git/hooks/post-update.sample
.git/hooks/pre-applypatch.sample
.git/hooks/pre-commit.sample
.git/hooks/pre-merge-commit.sample
.git/hooks/pre-push.sample
.git/hooks/pre-receive.sample
.git/hooks/push-to-checkout.sample
.git/hooks/update.sample
.git/hooks/fsmonitor-watchman.sample
.git/hooks/pre-rebase.sample
.git/hooks/prepare-commit-msg.sample
.git/info
.git/info/exclude
.git/description
.git/refs
.git/refs/heads
.git/refs/heads/master
.git/refs/tags
.git/refs/remotes
.git/refs/remotes/origin
.git/refs/remotes/origin/HEAD
.git/HEAD
.git/config
.git/objects
.git/objects/pack
.git/objects/pack/pack-772e37d34805d3ed99728898fecdb245d13221ae.pack
.git/objects/pack/pack-772e37d34805d3ed99728898fecdb245d13221ae.idx
.git/objects/info
.git/packed-refs
.git/logs
.git/logs/refs
.git/logs/refs/remotes
.git/logs/refs/remotes/origin
.git/logs/refs/remotes/origin/HEAD
.git/logs/refs/heads
.git/logs/refs/heads/master
.git/logs/HEAD
.git/index
.git/FETCH_HEAD
HEAD is now at eb8903969 FIX: st_dev field data type is unsigned int (Opcenter\Filesystem) CHG: workaround for blkcg_iostat_update kernel panic (System\Cgroup)
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Nothing to install, update or remove
Package swiftmailer/swiftmailer is abandoned, you should avoid using it. Use symfony/mailer instead.
Package symfony/debug is abandoned, you should avoid using it. Use symfony/error-handler instead.
Package true/punycode is abandoned, you should avoid using it. No replacement was suggested.
Generating autoload files
90 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
WARNING: Cache_Mproxy::_connect(): cache: cannot access server
(RedisException) EXCEPTION: Redis server went away
[/usr/local/apnscp/lib/Cache/Mproxy.php:108]
0. Redis->setOption(1, 1)
[/usr/local/apnscp/lib/Cache/Mproxy.php:108]
1. Cache_Mproxy::init()
[/usr/local/apnscp/lib/Cache/Base.php:87]
2. Cache_Base::_init()
[/usr/local/apnscp/lib/Cache/Base.php:40]
3. Cache_Base::spawn()
[/usr/local/apnscp/lib/Module/Support/Webapps/PathManager.php:29]
4. Module\Support\Webapps\PathManager::applicationViewPaths()
[/usr/local/apnscp/lib/Lararia/Providers/AppServiceProvider.php:40]
5. Lararia\Providers\AppServiceProvider->register()
[/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/Application.php:616]
6. Illuminate\Foundation\Application->register(Lararia\Providers\AppServiceProvider)
[/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/ProviderRepository.php:75]
7. Illuminate\Foundation\ProviderRepository->load(["Illuminate\Auth\AuthServiceProvider", "Illuminate\Broadcasting\BroadcastServiceProvider", "Illuminate\Bus\BusServiceProvider", "Illuminate\Cache\CacheServiceProvider", "Illuminate\Foundation\Providers\ConsoleSupportServiceProvider", "Illuminate\Cookie\CookieServiceProvider", "Illuminate\Database\DatabaseServiceProvider", "Illuminate\Encryption\EncryptionServiceProvider", "Illuminate\Filesystem\FilesystemServiceProvider", "Illuminate\Foundation\Providers\FoundationServiceProvider", ])
[/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/Application.php:593]
8. Illuminate\Foundation\Application->registerConfiguredProviders()
[/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/RegisterProviders.php:17]
9. Illuminate\Foundation\Bootstrap\RegisterProviders->bootstrap(Lararia\Application)
[/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/Application.php:219]
10. Illuminate\Foundation\Application->bootstrapWith(["Illuminate\Foundation\Bootstrap\LoadEnvironmentVariables", "Illuminate\Foundation\Bootstrap\LoadConfiguration", "Illuminate\Foundation\Bootstrap\HandleExceptions", "Illuminate\Foundation\Bootstrap\RegisterFacades", "Illuminate\Foundation\Bootstrap\SetRequestForConsole", "Illuminate\Foundation\Bootstrap\RegisterProviders", "Illuminate\Foundation\Bootstrap\BootProviders"])
[/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:320]
11. Illuminate\Foundation\Console\Kernel->bootstrap()
[/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129]
12. Illuminate\Foundation\Console\Kernel->handle(Symfony\Component\Console\Input\ArgvInput, Symfony\Component\Console\Output\ConsoleOutput)
[/usr/local/apnscp/lib/Lararia/Console/Kernel.php:31]
13. Lararia\Console\Kernel->handle(Symfony\Component\Console\Input\ArgvInput, Symfony\Component\Console\Output\ConsoleOutput)
[/usr/local/apnscp/artisan:69]
(RedisException) EXCEPTION: Redis server went away
[/usr/local/apnscp/lib/Cache/Mproxy.php:108]
0. Redis->setOption(1, 1)
[/usr/local/apnscp/lib/Cache/Mproxy.php:108]
1. Cache_Mproxy::init()
[/usr/local/apnscp/lib/Cache/Base.php:87]
2. Cache_Base::_init()
[/usr/local/apnscp/lib/Cache/Base.php:40]
3. Cache_Base::spawn()
[/usr/local/apnscp/lib/Module/Support/Webapps/PathManager.php:29]
4. Module\Support\Webapps\PathManager::applicationViewPaths()
[/usr/local/apnscp/lib/Lararia/Providers/AppServiceProvider.php:40]
5. Lararia\Providers\AppServiceProvider->register()
[/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/Application.php:616]
6. Illuminate\Foundation\Application->register(Lararia\Providers\AppServiceProvider)
[/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/ProviderRepository.php:75]
7. Illuminate\Foundation\ProviderRepository->load(["Illuminate\Auth\AuthServiceProvider", "Illuminate\Broadcasting\BroadcastServiceProvider", "Illuminate\Bus\BusServiceProvider", "Illuminate\Cache\CacheServiceProvider", "Illuminate\Foundation\Providers\ConsoleSupportServiceProvider", "Illuminate\Cookie\CookieServiceProvider", "Illuminate\Database\DatabaseServiceProvider", "Illuminate\Encryption\EncryptionServiceProvider", "Illuminate\Filesystem\FilesystemServiceProvider", "Illuminate\Foundation\Providers\FoundationServiceProvider", ])
[/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/Application.php:593]
8. Illuminate\Foundation\Application->registerConfiguredProviders()
[/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/RegisterProviders.php:17]
9. Illuminate\Foundation\Bootstrap\RegisterProviders->bootstrap(Lararia\Application)
[/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/Application.php:219]
10. Illuminate\Foundation\Application->bootstrapWith(["Illuminate\Foundation\Bootstrap\LoadEnvironmentVariables", "Illuminate\Foundation\Bootstrap\LoadConfiguration", "Illuminate\Foundation\Bootstrap\HandleExceptions", "Illuminate\Foundation\Bootstrap\RegisterFacades", "Illuminate\Foundation\Bootstrap\SetRequestForConsole", "Illuminate\Foundation\Bootstrap\RegisterProviders", "Illuminate\Foundation\Bootstrap\BootProviders"])
[/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:320]
11. Illuminate\Foundation\Console\Kernel->bootstrap()
[/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129]
12. Illuminate\Foundation\Console\Kernel->handle(Symfony\Component\Console\Input\ArgvInput, Symfony\Component\Console\Output\ConsoleOutput)
[/usr/local/apnscp/lib/Lararia/Console/Kernel.php:31]
13. Lararia\Console\Kernel->handle(Symfony\Component\Console\Input\ArgvInput, Symfony\Component\Console\Output\ConsoleOutput)
[/usr/local/apnscp/artisan:69]
No, remove apnscp-2/ first before redoing this.
im not sure where and how?
it was in the up dir. sorry.
HEAD is now at eb8903969 FIX: st_dev field data type is unsigned int (Opcenter\Filesystem) CHG: workaround for blkcg_iostat_update kernel panic (System\Cgroup)
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Nothing to install, update or remove
Package swiftmailer/swiftmailer is abandoned, you should avoid using it. Use symfony/mailer instead.
Package symfony/debug is abandoned, you should avoid using it. Use symfony/error-handler instead.
Package true/punycode is abandoned, you should avoid using it. No replacement was suggested.
Generating autoload files
90 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
Nothing to migrate.
Nothing to migrate.
i can now login at the webmail.
and the CP is up and working again!
<3 thanks Matt!
Can u tell me what was wrong? or did i do something not ok?
Storage corruption or the server ran out of storage.
storage curruption? can i check that in someway? or whats the best resolution for this?
50% disk space left…
df. On a multi partition layout account data is within /home, panel data is within /.
/dev/mapper/rl_cloud-root 73364480 21042100 52322380 29% /
and
/dev/mapper/rl_cloud-home 654405432 129675168 524730264 20% /home