Directly after installation: apnscp.service start operation timed out. Terminating.

Fresh installation on a latest CentOS VM. Did a full yum-update prior to installation, a yum-update after installation found nothing to update. Attempting to connect to localhost:2082 failed.

Installed apnscp with all the maximums: MariaDB 10.4, Postgres 11, PHP 7.3. Used standard “admin” username, employed a functional eMail address.

From the end of the apnscp-bootstraper.log:

-- The start-up result is done.
Jul 30 08:01:01 kabis.dev CROND[176644]: (root) CMD (run-parts /etc/cron.hourly)
Jul 30 08:01:01 kabis.dev run-parts(/etc/cron.hourly)[176647]: starting 0anacron
Jul 30 08:01:01 kabis.dev run-parts(/etc/cron.hourly)[176653]: finished 0anacron
Jul 30 08:01:01 kabis.dev run-parts(/etc/cron.hourly)[176655]: starting 0yum-hourly.cron
Jul 30 08:01:01 kabis.dev run-parts(/etc/cron.hourly)[176659]: finished 0yum-hourly.cron
Jul 30 08:01:01 kabis.dev systemd[1]: Removed slice User Slice of root.
-- Subject: Unit user-0.slice has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit user-0.slice has finished shutting down.
Jul 30 08:58:15 kabis.dev polkitd[905]: Registered Authentication Agent for unix-process:176848:3021506 (system bus name :1.56 [<unknown>], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_CA.UTF-8)
Jul 30 08:58:15 kabis.dev systemd[1]: Cannot add dependency job for unit proc-sys-fs-binfmt_misc.automount, ignoring: Unit is masked.
Jul 30 08:58:15 kabis.dev systemd[1]: Cannot add dependency job for unit systemd-binfmt.service, ignoring: Unit is masked.
Jul 30 08:58:15 kabis.dev systemd[1]: Starting apnscp...
-- Subject: Unit apnscp.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit apnscp.service has begun starting up.
Jul 30 08:58:25 kabis.dev systemd[1]: apnscp.service start operation timed out. Terminating.
Jul 30 08:58:25 kabis.dev systemd[1]: Failed to start apnscp.
-- Subject: Unit apnscp.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit apnscp.service has failed.
--
-- The result is failed.
Jul 30 08:58:25 kabis.dev systemd[1]: Unit apnscp.service entered failed state.
Jul 30 08:58:25 kabis.dev systemd[1]: apnscp.service failed.
Jul 30 08:58:25 kabis.dev polkitd[905]: Unregistered Authentication Agent for unix-process:176848:3021506 (system bus name :1.56, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_CA.UTF-8) (disconnected fr
Jul 30 08:58:26 kabis.dev apnscp.init[176854]: [42B blob data]

When looking into the apnscp service:

# systemctl status apnscp.service
● apnscp.service - apnscp
   Loaded: loaded (/etc/systemd/system/apnscp.service; enabled; vendor preset: disabled)
   Active: failed (Result: timeout) since Tue 2019-07-30 19:43:43 PDT; 2min 53s ago
  Process: 11100 ExecStop=/etc/systemd/user/apnscp.init stop (code=exited, status=0/SUCCESS)
 Main PID: 3233 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/apnscp.service
           ├─11139 /usr/sbin/httpd -f /usr/local/apnscp/config/httpd.conf -k start
           ├─11144 /usr/sbin/httpd -f /usr/local/apnscp/config/httpd.conf -k start
           ├─11146 /usr/sbin/httpd -f /usr/local/apnscp/config/httpd.conf -k start
           ├─11259 /usr/local/apnscp/apps/terminal/shellinabox/bin/shellinaboxd -t --background=/usr/local/apnscp/storage/run/shellinabox.pid -s /:SSH:localhost --no-beep --localhost-only --user=root
           ├─11262 /usr/local/apnscp/apps/terminal/shellinabox/bin/shellinaboxd -t --background=/usr/local/apnscp/storage/run/shellinabox.pid -s /:SSH:localhost --no-beep --localhost-only --user=root
           ├─11506 apnscpd: master
           ├─11519 job runner
           ├─11520 apnscpd: idle
           ├─11545 /usr/local/apnscp/bin/php-bins/apnscp_php /usr/local/apnscp/artisan horizon
           ├─11554 /usr/local/apnscp/bin/php-bins/apnscp_php artisan horizon:supervisor kabisdev-tAVu:supervisor-1 redis --delay=0 --memory=128 --queue=high,default --sleep=3 --timeout=60 --tries=3 --balance= --max-processes...
           └─11565 /usr/local/apnscp/bin/php-bins/apnscp_php artisan horizon:work redis --delay=0 --memory=128 --queue=high,default --sleep=3 --timeout=60 --tries=3 --supervisor=kabisdev-tAVu:supervisor-1

Jul 30 19:43:33 kabis.dev systemd[1]: Starting apnscp...
Jul 30 19:43:43 kabis.dev systemd[1]: apnscp.service start operation timed out. Terminating.
Jul 30 19:43:43 kabis.dev systemd[1]: Failed to start apnscp.
Jul 30 19:43:43 kabis.dev systemd[1]: Unit apnscp.service entered failed state.
Jul 30 19:43:43 kabis.dev systemd[1]: apnscp.service failed.
Jul 30 19:43:43 kabis.dev apnscp.init[11495]: [42B blob data]

Honestly don’t know what I need to be looking for. Ran into this forum, double-checked a few similar posts, but didn’t find anything that helped.

tail -n 20 /usr/local/apnscp/storage/logs/start.log

What output do you get from there? Alternatively,

cd /usr/local/apnscp/bin
apnscpd -f restart

You may need to run that twice if it systemd kills it the first time. Since the state is failed it shouldn’t.

Also,

grep -B20 -i recap /root/apnscp-bootstrapper.log

That’ll give the last 20 lines prior to the conclusion of Bootstrapper. This should be enough starting points to work off of.

Interesting. Your first command:

[Tue Jul 30 22:43:44 2019] Updating browscap
[Tue Jul 30 22:43:44 2019] Rewriting AOF data
ERROR   : fail2ban-client: ERROR  Failed to access socket path: /var/run/fail2ban/fail2ban.sock. Is fail2ban running?

ERROR   : failed to query fail2ban, `ERROR  Failed to access socket path: /var/run/fail2ban/fail2ban.sock. Is fail2ban running?
'
WARNING : error retrieving fail2ban jail configuration
[Tue Jul 30 22:43:44 2019] Loading terminal...
WARNING: copy(/home/virtual/FILESYSTEMTEMPLATE/siteinfo//usr/share/pear/wp-cli.phar): failed to open stream: No such file or directory
[/usr/local/apnscp/lib/modules/wordpress.php:1195]
[Tue Jul 30 22:43:44 2019] WARN: housekeeping problem on `wordpress'
[Tue Jul 30 22:43:44 2019] Spawning worker cron
[Tue Jul 30 22:43:44 2019] Spawning worker
[Tue Jul 30 22:43:44 2019] Initiating cron
Horizon started successfully.
WARNING : Job worker died (PID 11274) - restarting in 30 seconds. Count 1
WARNING: socket_select(): unable to select [4]: Interrupted system call
[/usr/local/apnscp/lib/lservicelib.pht:858]
[Tue Jul 30 22:44:16 2019] apnscpd restart. Shutting down job runner
Horizon started successfully.

Your second suggestion:

bash: apnscpd: command not found

And yes, I did go to that directory. Successfully. Despite this bash error, an ls -l clearly shows the file:

total 32
-rwxr-xr-x. 1 apnscp apnscp 1145 2019-07-30 00:24 ActivateDomain
-rwxr-xr-x. 1 apnscp apnscp 1232 2019-07-30 00:24 AddDomain
-rwxr-xr-x. 1 apnscp apnscp 2129 2019-07-30 00:24 apnscpd
-rwxr-xr-x. 1 apnscp apnscp  117 2019-07-30 00:24 cmd
-rwxr-xr-x. 1 apnscp apnscp 1789 2019-07-30 00:24 DeleteDomain
-rwxr-xr-x. 1 apnscp apnscp 1628 2019-07-30 00:24 EditDomain
drwxr-xr-x. 2 apnscp apnscp   76 2019-07-30 00:24 hooks
drwxr-xr-x. 2 apnscp apnscp  153 2019-07-30 01:45 php-bins
drwxr-xr-x. 2 apnscp apnscp 4096 2019-07-30 00:24 scripts
-rwxr-xr-x. 1 apnscp apnscp 1144 2019-07-30 00:24 SuspendDomain

And I had already elevated myself into su, so this wasn’t a permissions issue.

Your third suggestion:

2019-07-30 19:30:03,282 p=3230 u=root |  TASK [apnscp/initialize-db : Verify apnscp grants] *****************************
2019-07-30 19:30:03,901 p=3230 u=root |  ok: [localhost]
2019-07-30 19:30:03,959 p=3230 u=root |  TASK [include_role : common/merge-yaml-config] *********************************
2019-07-30 19:30:03,991 p=3230 u=root |  skipping: [localhost]
2019-07-30 19:30:04,056 p=3230 u=root |  TASK [apnscp/create-admin : stat] **********************************************
2019-07-30 19:30:04,263 p=3230 u=root |  ok: [localhost]
2019-07-30 19:30:04,323 p=3230 u=root |  TASK [apnscp/create-admin : copy] **********************************************
2019-07-30 19:30:04,692 p=3230 u=root |  changed: [localhost]
2019-07-30 19:30:04,750 p=3230 u=root |  TASK [apnscp/create-admin : Start apnscp] **************************************
2019-07-30 19:30:15,187 p=3230 u=root |  changed: [localhost]
2019-07-30 19:30:15,247 p=3230 u=root |  TASK [apnscp/create-admin : set_fact] ******************************************
2019-07-30 19:30:15,281 p=3230 u=root |  ok: [localhost]
2019-07-30 19:30:15,341 p=3230 u=root |  TASK [apnscp/create-admin : Setting admin password] ****************************
2019-07-30 19:30:19,058 p=3230 u=root |  changed: [localhost]
2019-07-30 19:30:19,117 p=3230 u=root |  TASK [apnscp/create-admin : debug] *********************************************
2019-07-30 19:30:19,147 p=3230 u=root |  skipping: [localhost]
2019-07-30 19:30:19,222 p=3230 u=root |  TASK [apnscp/create-admin : stat] **********************************************
2019-07-30 19:30:19,410 p=3230 u=root |  ok: [localhost]
2019-07-30 19:30:19,468 p=3230 u=root |  TASK [apnscp/create-admin : Restart apnscp housekeeper] ************************
2019-07-30 19:30:41,276 p=3230 u=root |  fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unable to restart service apnscp: Job for apnscp.service failed because a timeout was exceeded. See \"systemctl status apnscp.service\" and \"journalctl -xe\" for details.\n"}
2019-07-30 19:30:41,279 p=3230 u=root |  PLAY RECAP *********************************************************************

apnscp/create-admin happens early in the Bootstrap process. I’m building out on a Vultr machine and well past that stage into apnscp/initialize-filesystem-template.

Take a look at /usr/local/apnscp/storage/logs/start.log or just email (matt@apisnetworks.com) or PM me the log. A restart failed and caused Bootstrapper to terminate, but there’s not enough information to determine why.

My start.log:

WARNING : You are running a bugged version of the Linux kernel. Workarounds in place. Upgrade to a kernel newer than 3.10.1
WARNING : Filesystem virtual root `/home/virtual/FILESYSTEMTEMPLATE' uses a different mount point. Duplicating files will double storage usage in this setup. / and `/home/virtual/FILESYSTEMTEMPLATE' should use the same filesystem mount.
[Tue Jul 30 04:53:12 2019] Checking if frontend started
INFO    : Frontend started with pid `175253'
[Tue Jul 30 04:53:12 2019] Loading modules...
[Tue Jul 30 04:53:12 2019] + Loaded module admin in 0.02328 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module aliases in 0.01778 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module argos in 0.00173 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module auth in 0.00349 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module bandwidth in 0.00070 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module billing in 0.00077 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module cgroup in 0.00148 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module cloudflare in 0.00181 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module common in 0.00238 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module config in 0.00089 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module crm in 0.00065 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module crontab in 0.00189 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module dav in 0.00036 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module demo in 0.00051 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module discourse in 0.00529 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module diskquota in 0.00068 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module dns in 0.00521 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module drupal in 0.00280 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module email in 0.00578 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module file in 0.00618 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module frontpage in 0.00089 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module ftp in 0.00140 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module ghost in 0.00213 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module git in 0.00064 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module htpasswd in 0.00058 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module ipinfo in 0.00077 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module joomla in 0.00222 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module laravel in 0.00234 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module letsencrypt in 0.00345 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module log in 0.00058 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module logs in 0.00133 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module magento in 0.00269 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module majordomo in 0.00225 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module misc in 0.00085 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module mysql in 0.00427 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module node in 0.00069 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module perl in 0.00044 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module personality in 0.00103 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module pgsql in 0.00323 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module php in 0.00182 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module pman in 0.00092 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module python in 0.00043 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module quote in 0.00058 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module rampart in 0.00096 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module redis in 0.00077 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module reseller in 0.00037 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module ruby in 0.00084 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module site in 0.00111 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module spamfilter in 0.00044 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module sql in 0.00129 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module ssh in 0.00064 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module ssl in 0.00212 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module stats in 0.00127 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module telemetry in 0.00050 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module test in 0.00126 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module tomcat in 0.00076 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module user in 0.00307 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module verco in 0.00081 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module watch in 0.00086 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module webhook in 0.00054 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module web in 0.00240 seconds
[Tue Jul 30 04:53:12 2019] + Loaded module wordpress in 0.00371 seconds
[Tue Jul 30 04:53:12 2019] Generating IntelliSense...
[Tue Jul 30 04:53:12 2019] Generating WSDL...
[Tue Jul 30 04:53:12 2019] WARNING: Unrecognized return type \Opcenter\Dns\Record for Dns_Module::provisioning_records
[Tue Jul 30 04:53:12 2019] Housekeeping...

Not too sure why it’s complaining about my kernel, I have the latest… (at least, as of the 19th of July):

3.10.0-957.21.3.el7.x86_64

There should be more in start.log; is that the entirety of it? Kernel complaint is idiosyncratic to RHEL7. If you run a custom kernel (set custom_kernel_rpm to kernel-ml or kernel-lt) it’ll go away. It’s a workaround with remounting OverlayFS on those versions. Who’s your provider or is this on bare metal?

Confirmed: that is the entire start.log.

This is a Hyper-V VM on Windows Server Datacentre 2016, with CentOS 7 (latest) installed. My own iron, my own hosting. I made doubly sure to apply all updates before installing apnscp, and I rebooted for good measure. This machine is on an internal network with a private static IP, but will eventually be connected to the Internet for various purposes.

About the only thing I did different than a butt-normal install of CentOS was with the partitioning and the security profile during insall. Here are my partitions:

Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/cl_kabis-root     4.0G  130M  3.9G   4% /
devtmpfs                      1.9G     0  1.9G   0% /dev
tmpfs                         1.9G  8.0K  1.9G   1% /dev/shm
tmpfs                         1.9G   26M  1.9G   2% /run
tmpfs                         1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mapper/cl_kabis-usr       20G  4.7G   15G  24% /usr
/dev/sda2                     250M  157M   93M  63% /boot
/dev/sda1                      64M   12M   53M  18% /boot/efi
tmpfs                         938M   44K  938M   1% /tmp
/dev/mapper/cl_kabis-home     4.0G   33M  4.0G   1% /home
/dev/mapper/cl_kabis-opt      2.0G   33M  2.0G   2% /opt
/dev/mapper/cl_kabis-var      8.0G  1.3G  6.8G  16% /var
/dev/mapper/cl_kabis-var_log  8.0G   39M  8.0G   1% /var/log
/dev/mapper/cl_kabis-var_db   8.0G   33M  8.0G   1% /var/db
/dev/mapper/cl_kabis-var_www   48G   33M   48G   1% /var/www
tmpfs                         376M     0  376M   0% /run/user/988
tmpfs                         376M     0  376M   0% /run/user/1000

And the security profile was the standard server package second from the top; nothing strange. My partitioning mania is due to my BSD heritage; I have learned to hate log bombs and such.

Out of curiosity,

cd /usr/local/apnscp/resources/playbooks
ansible-playbook bootstrap.yml

How’s that work for you? Also, would recommend standard partitions. ulimit will set a clamp on max filesize which is a good technique to avoid logs running amok. It could be an issue between having /, /var, and /usr on different partitions. I know for example apnscp installed under /usr/local/apnscp will hardlink the SSO webmail utility to /var/www/html. You can’t cross-link devices, so there’s one aspect of the panel that’ll fail with this partition layout.

Put / and, if you want, /home on its own partition then run the installation again. I keep them all on the same mount FWIW. This is not an advisable layout, especially with regards to storage allocation unless each of those mounts are thinly provisioned via LVM.

When I ran your command, I could only paste a certain amount in due to a character limitation, so I grabbed the last bit:

TASK [apnscp/bootstrap : Setting dns => recursive_ns = 1.0.0.1,1.1.1.1] ***********************************************************************************************************************************************************
skipping: [localhost]

TASK [apnscp/bootstrap : Change apnscp ownership] *********************************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/build-php : include_vars] ********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/build-php : Install apnscp build RPMs] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/build-php : Verify extension_dir setting] ****************************************************************************************************************************************************************************
ok: [localhost]

TASK [include_role : php/build-from-source] ***************************************************************************************************************************************************************************************

TASK [php/build-from-source : Download PHP release feed] **************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/build-from-source : set_fact] *******************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/build-from-source : set_fact] *******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [include_role : php/build-from-source] ***************************************************************************************************************************************************************************************

TASK [php/build-from-source : stat] ***********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/build-from-source : stat] ***********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/build-from-source : stat] ***********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/build-from-source : command] ********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/build-from-source : set_fact] *******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/build-from-source : Cleanup older PHPs] *********************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/build-from-source : Remove older PHPs] **********************************************************************************************************************************************************************************

TASK [php/build-from-source : set_fact] *******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/build-from-source : Download PHP 7.2.20] ********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/build-from-source : Compile PHP 7.2.20] *********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/build-from-source : Clean PHP code] *************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/build-from-source : Verify clamd stopped (OOM)] *************************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/build-from-source : make] ***********************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/build-from-source : command] ********************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/build-from-source : make] ***********************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/build-from-source : file] ***********************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/build-from-source : set_fact] *******************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [apnscp/build-php : file] ****************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/install-extensions : include_vars] ***********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [apnscp/install-extensions : set_fact] ***************************************************************************************************************************************************************************************
ok: [localhost]

TASK [include_role : php/common] **************************************************************************************************************************************************************************************************

TASK [php/common : Query PHP API version] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/common : set_fact] ******************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [include_role : php/install-pecl-module] *************************************************************************************************************************************************************************************

TASK [php/install-pecl-module : include_tasks] ************************************************************************************************************************************************************************************
included: /usr/local/apnscp/resources/playbooks/roles/php/install-pecl-module/tasks/verify-and-install.yml for localhost
included: /usr/local/apnscp/resources/playbooks/roles/php/install-pecl-module/tasks/verify-and-install.yml for localhost
included: /usr/local/apnscp/resources/playbooks/roles/php/install-pecl-module/tasks/verify-and-install.yml for localhost

TASK [include_role : php/common] **************************************************************************************************************************************************************************************************

TASK [php/common : Query PHP API version] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/common : set_fact] ******************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : command] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : command] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : set_fact] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Update channel protocols] *************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Verify redis module installed] ********************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Build and install redis library.] *****************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/install-pecl-module : Check if redis loaded] ****************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Bypass individual ini file creation for redis] ****************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Verify /usr/local/apnscp/config/php.d/redis.ini exists] *******************************************************************************************************************************************
skipping: [localhost]

TASK [php/install-pecl-module : Add PECL module configuration] ********************************************************************************************************************************************************************
skipping: [localhost]

TASK [include_role : php/common] **************************************************************************************************************************************************************************************************

TASK [php/common : Query PHP API version] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/common : set_fact] ******************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : command] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : command] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : set_fact] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Update channel protocols] *************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Verify igbinary module installed] *****************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Build and install igbinary library.] **************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/install-pecl-module : Check if igbinary loaded] *************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Bypass individual ini file creation for igbinary] *************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Verify /usr/local/apnscp/config/php.d/igbinary.ini exists] ****************************************************************************************************************************************
skipping: [localhost]

TASK [php/install-pecl-module : Add PECL module configuration] ********************************************************************************************************************************************************************
skipping: [localhost]

TASK [include_role : php/common] **************************************************************************************************************************************************************************************************

TASK [php/common : Query PHP API version] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/common : set_fact] ******************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : command] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : command] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : set_fact] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Update channel protocols] *************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Verify apcu module installed] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Build and install apcu library.] ******************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/install-pecl-module : Check if apcu loaded] *****************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Bypass individual ini file creation for apcu] *****************************************************************************************************************************************************
ok: [localhost]

TASK [php/install-pecl-module : Verify /usr/local/apnscp/config/php.d/apcu.ini exists] ********************************************************************************************************************************************
skipping: [localhost]

TASK [php/install-pecl-module : Add PECL module configuration] ********************************************************************************************************************************************************************
skipping: [localhost]

TASK [Install ionCube] ************************************************************************************************************************************************************************************************************

TASK [Lookup PHP 7.2 extension dir] ***********************************************************************************************************************************************************************************************

TASK [php/common : Query PHP API version] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/common : set_fact] ******************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install : stat] *********************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install : file] *********************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [php/install : set_fact] *****************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [include_role : php/ioncube-loader] ******************************************************************************************************************************************************************************************

TASK [php/ioncube-loader : Include platform specific variables] *******************************************************************************************************************************************************************
ok: [localhost]

TASK [php/ioncube-loader : Define php_extension_conf_paths] ***********************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/ioncube-loader : Define php_ioncube_loader_module_path] *****************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/ioncube-loader : Define php_ioncube_loader_module_filename] *************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/ioncube-loader : Define php_ioncube_loader_config_filename] *************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/ioncube-loader : Download and untar ioncube-loader] *********************************************************************************************************************************************************************
skipping: [localhost]

TASK [php/ioncube-loader : Ensure ioncube-loader module path exists] **************************************************************************************************************************************************************
ok: [localhost]

TASK [php/ioncube-loader : Move ioncube-loader module into place] *****************************************************************************************************************************************************************
ok: [localhost]

TASK [php/ioncube-loader : Copy ioncube-loader ini into main extension config folder] *********************************************************************************************************************************************
ok: [localhost] => (item=/usr/local/apnscp/config/php.d)

TASK [apnscp/install-extensions : Load opcache into php.ini] **********************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/install-vendor-library : command] ************************************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/install-vendor-library : composer] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/install-vendor-library : composer] ***********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [apnscp/install-vendor-library : command] ************************************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/install-vendor-library : Install vendor libraries] *******************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/initialize-db : set_fact] ********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/initialize-db : stat] ************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/initialize-db : set_fact] ********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/initialize-db : set_fact] ********************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [apnscp/initialize-db : set_fact] ********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/initialize-db : file] ************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [apnscp/initialize-db : Verify apnscp database exists] ***********************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/initialize-db : Create apnscp database] ******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [apnscp/initialize-db : Verify apnscp user exists] ***************************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/initialize-db : Create apnscp user] **********************************************************************************************************************************************************************************
ok: [localhost]

TASK [include_role : common/merge-yaml-config] ************************************************************************************************************************************************************************************
skipping: [localhost] => (item=api)
skipping: [localhost] => (item=apnscp)
skipping: [localhost] => (item=auth)
skipping: [localhost] => (item=billing)
skipping: [localhost] => (item=crm)
skipping: [localhost] => (item=dns)

TASK [apnscp/initialize-db : Verify apnscp user exists] ***************************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/initialize-db : Verify appldb database exists] ***********************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/initialize-db : tempfile] ********************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [apnscp/initialize-db : template] ********************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [apnscp/initialize-db : postgresql_db] ***************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [apnscp/initialize-db : file] ************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [apnscp/initialize-db : Verify apnscp grants] ********************************************************************************************************************************************************************************
ok: [localhost]

TASK [include_role : common/merge-yaml-config] ************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [apnscp/create-admin : stat] *************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [apnscp/create-admin : copy] *************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [apnscp/create-admin : Start apnscp] *****************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => changed=false
  msg: |-
    Unable to start service apnscp: Job for apnscp.service failed because a timeout was exceeded. See "systemctl status apnscp.service" and "journalctl -xe" for details.

PLAY RECAP ************************************************************************************************************************************************************************************************************************
localhost                  : ok=342  changed=0    unreachable=0    failed=1    skipped=191  rescued=0    ignored=0

And I should have mentioned: yes, all partitions are thinly provisioned. That’s SOP with any VM.

It’s a bit late to nuke-and-rebuild this evening, but if you can’t spot anything else I will do exactly that later this week. I am not comfortable with any system where the partitions aren’t properly segregated, but I will trust your judgement in this matter. As you pointed out, there is at least one part that will fail if a file system is securely partitioned.

At first blush housekeeping is failing to link webmail SSO dispatcher to /var/www/html.

I’d nuke. /var/db isn’t used on RedHat. /var/lib/mysql and /var/lib/pgsql are the data directories for the respective RDBMS, so you’d be constrained to the 8 GB layout shared by other services under /var/cache, /var/tmp, etc.

This is from the licensing server for example,

24      ./spool
856     ./db
45072   ./log
163952  ./www
1174428 ./cache
1483060 ./lib

And from a server with around 450 domains,

1072     ./db
81448    ./spool
164276   ./www
3074548  ./log
7595264  ./lib
15134860 ./cache