hundrets or thousands of /home/virtual/siteXY created

Support
1

I don’t really know what is the problem but I just found out that my /home/virtual/ directory contains an unreal amount of sites. Nexus has no sites at all configured on this host.

ls -la /home/virtual/
Display all 1817 possibilities? (y or n)

lrwxrwxrwx    1 root root    25  9. Nov 13:19 admin728 -> /home/virtual/site728/fst
lrwxrwxrwx    1 root root    25  9. Nov 13:22 admin729 -> /home/virtual/site729/fst
lrwxrwxrwx    1 root root    25  9. Nov 13:23 admin730 -> /home/virtual/site730/fst
lrwxrwxrwx    1 root root    25  9. Nov 13:24 admin731 -> /home/virtual/site731/fst
lrwxrwxrwx    1 root root    25  9. Nov 13:26 admin732 -> /home/virtual/site732/fst
lrwxrwxrwx    1 root root    25  9. Nov 13:30 admin733 -> /home/virtual/site733/fst
lrwxrwxrwx    1 root root    25  9. Nov 13:31 admin734 -> /home/virtual/site734/fst
...
lrwxrwxrwx    1 root root    25  9. Nov 11:06 apiscp-int-apasbnjlkddynhqk.test -> /home/virtual/site656/fst
lrwxrwxrwx    1 root root    25  9. Nov 05:52 apiscp-int-apuapbgbwyccbyay.test -> /home/virtual/site434/fst
lrwxrwxrwx    1 root root    25  9. Nov 08:31 apiscp-int-apyvpadefbaupdmi.test -> /home/virtual/site561/fst
lrwxrwxrwx    1 root root    25  9. Nov 03:03 apiscp-int-aqakmzzsoicwayoi.test -> /home/virtual/site289/fst
lrwxrwxrwx    1 root root    25  9. Nov 02:58 apiscp-int-aqaqdhljibrhyoed.test -> /home/virtual/site285/fst
lrwxrwxrwx    1 root root    25  9. Nov 10:48 apiscp-int-aquhgwhlbklesahq.test -> /home/virtual/site646/fst
lrwxrwxrwx    1 root root    25  9. Nov 08:36 apiscp-int-aqznwbsqrdrhnctq.test -> /home/virtual/site565/fst
lrwxrwxrwx    1 root root    25  9. Nov 13:43 apiscp-int-aydabvayeewicgky.test -> /home/virtual/site740/fst
lrwxrwxrwx    1 root root    25  9. Nov 06:38 apiscp-int-ayiqshfnbobpikep.test -> /home/virtual/site470/fst
...
drwx--x--x    3 root root    20 23. Nov 11:09 site232
drwx--x--x    3 root root    20 23. Nov 11:09 site233
drwx--x--x    3 root root    20 23. Nov 11:09 site234
drwx--x--x    3 root root    20 23. Nov 11:09 site235
drwx--x--x    3 root root    20 23. Nov 11:09 site236
drwx--x--x    3 root root    20 23. Nov 11:09 site237
drwx--x--x    3 root root    20 23. Nov 11:09 site238

The softlinks are all dead.
Can I just clean the entire /home/virtual/ folder? How to prevent it from happening again?

This is a different host than the one with the LoadConfiguration issue btw.

2

Internal check is failing. This is related to your existing issue, LoadConfiguration: must be of type array, int given - #10 by msaladna

Let’s focus on that. Once resolved,

cd /home/virtual
DeleteDomain --force apiscp-int-*
5

Thanks for reopening.

Executing the command you posted results in a long list of these kind of errors:

ERROR  : unknown(): failed to remove site `site669': fatal(): failed to generate user session `rdwjgtczqakqbrnl'@`apiscp-int-yqanftjhjecztlji.test'
         0. Error_Reporter::add_error("failed to remove site `%s': %s", ["site669", "fatal(): failed to generate user session `rdwjgtczqakqbrnl'@`apiscp-int-yqanftjhjecztlji.test'"])
            [/usr/local/apnscp/lib/log_wrapper.php:72]
         1. error("failed to remove site `%s': %s", "site669", "fatal(): failed to generate user session `rdwjgtczqakqbrnl'@`apiscp-int-yqanftjhjecztlji.test'")
            [/usr/local/apnscp/bin/DeleteDomain:99]
ERROR  : Auth_Info_User::initializeUser(): apiscp-int-yoohldwjlgnmoegk.test: cannot access /etc/passwd
         0. Error_Reporter::add_error("apiscp-int-yoohldwjlgnmoegk.test: cannot access /etc/passwd", )
            [/usr/local/apnscp/lib/log_wrapper.php:72]
         1. error("apiscp-int-yoohldwjlgnmoegk.test: cannot access /etc/passwd")
            [/usr/local/apnscp/lib/Auth/Info/User.php:174]
         2. Auth_Info_User::initializeUser("bbkqhuiilzvfqsrt", "apiscp-int-yoohldwjlgnmoegk.test", null, "RQDLdPPY5WQrdDZzscupoJPh9mPcTp6E")
            [/usr/local/apnscp/lib/Auth.php:847]
         3. Auth->initializeUser("bbkqhuiilzvfqsrt", "apiscp-int-yoohldwjlgnmoegk.test", null)
            [/usr/local/apnscp/lib/Auth/CLI.php:49]
         4. Auth_CLI->authenticate()
            [/usr/local/apnscp/lib/CLI/cmd.php:636]
         5. CLI\cmd("bbkqhuiilzvfqsrt", "apiscp-int-yoohldwjlgnmoegk.test")
            [/usr/local/apnscp/bin/DeleteDomain:91]
Unauthorized
6

Run /usr/local/apnscp/bin/scripts/mapCheck.php rebuild then reattempt. May take a couple iterations to clear itself up.

1 Like
7

Thanks. This script was running several minutes and spammed the terminal but now everything is cleaned up in /home/. Just site1 is remaining for some reason.
Maybe a bit too much has been deleted now. upcp --reset gives some error:

TASK [mail/configure-postfix : {{ item.value | default(false, true) | ternary('Set', 'Remove') }} {{item.key}} => {{ item.value}} in {{ config_file }}] ***
failed: [localhost] (item=Set message_size_limit => 100000000) => changed=false
  ansible_loop_var: item
  item:
    key: message_size_limit
    value: '100000000'
  msg: Destination /home/virtual/FILESYSTEMTEMPLATE/siteinfo//etc/postfix/main.cf does not exist !
  rc: 257

PLAY RECAP *********************************************************************
localhost                  : ok=7    changed=0    unreachable=0    failed=1    skipped=3    rescued=0    ignored=0



STDERR:
[/usr/local/apnscp/lib/Lararia/Database/Migrations/PlatformMigrator.php:211]

         0. Lararia\Database\Migrations\PlatformMigrator->runMigration("2024_11_14_162936_platform-tweaks", "up", [step:true])
            [/usr/local/apnscp/lib/Lararia/Database/Migrations/PlatformMigrator.php:128]
         1. Lararia\Database\Migrations\PlatformMigrator->runUp("/usr/local/apnscp/resources/playbooks/migrations/2024_11_14_162936_platform-tweaks.yml", 35, false)
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php:165]
         2. Illuminate\Database\Migrations\Migrator->runPending(["/usr/local/apnscp/resources/playbooks/migrations/2024_11_14_162936_platform-tweaks.yml"], [pretend:false, step:0, rollback:false, ff:false])
            [/usr/local/apnscp/lib/Lararia/Database/Migrations/PlatformMigrator.php:92]
         3. Lararia\Database\Migrations\PlatformMigrator->runPending(["/usr/local/apnscp/resources/playbooks/migrations/2024_11_14_162936_platform-tweaks.yml"], [pretend:false, step:0, rollback:false, ff:false])
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php:110]
         4. Illuminate\Database\Migrations\Migrator->run(["/usr/local/apnscp/resources/playbooks/migrations"], [pretend:false, step:0, rollback:false, ff:false])
            [/usr/local/apnscp/lib/Lararia/Database/Migrations/PlatformMigrator.php:40]
         5. Lararia\Database\Migrations\PlatformMigrator->run(["/usr/local/apnscp/resources/playbooks/migrations"], [pretend:false, step:0, rollback:false, ff:false])
            [/usr/local/apnscp/lib/Lararia/Console/Commands/PlatformMigration.php:76]
         6. Lararia\Console\Commands\PlatformMigration->handle()
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36]
         7. Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Container/Util.php:37]
         8. Illuminate\Container\Util::unwrapIfClosure()
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93]
         9. Illuminate\Container\BoundMethod::callBoundMethod(Lararia\Application, [], [Lararia\Console\Commands\PlatformMigration, "handle"], )
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37]
        10. Illuminate\Container\BoundMethod::call(Lararia\Application, [], [Lararia\Console\Commands\PlatformMigration, "handle"], null)
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Container/Container.php:590]
        11. Illuminate\Container\Container->call([Lararia\Console\Commands\PlatformMigration, "handle"])
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Console/Command.php:134]
        12. Illuminate\Console\Command->execute(Symfony\Component\Console\Input\ArrayInput, Illuminate\Console\OutputStyle)
            [/usr/local/apnscp/vendor/symfony/console/Command/Command.php:255]
        13. Symfony\Component\Console\Command\Command->run(Symfony\Component\Console\Input\ArrayInput, Illuminate\Console\OutputStyle)
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Console/Command.php:121]
        14. Illuminate\Console\Command->run(Symfony\Component\Console\Input\ArrayInput, Illuminate\Console\OutputStyle)
            [/usr/local/apnscp/vendor/symfony/console/Application.php:1021]
        15. Symfony\Component\Console\Application->doRunCommand(Lararia\Console\Commands\PlatformMigration, Symfony\Component\Console\Input\ArrayInput, Illuminate\Console\OutputStyle)
            [/usr/local/apnscp/vendor/symfony/console/Application.php:275]
        16. Symfony\Component\Console\Application->doRun(Symfony\Component\Console\Input\ArrayInput, Illuminate\Console\OutputStyle)
            [/usr/local/apnscp/vendor/symfony/console/Application.php:149]
        17. Symfony\Component\Console\Application->run(Symfony\Component\Console\Input\ArrayInput, Illuminate\Console\OutputStyle)
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Console/Application.php:93]
        18. Illuminate\Console\Application->run(Symfony\Component\Console\Input\ArrayInput, Illuminate\Console\OutputStyle)
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Console/Application.php:185]
        19. Illuminate\Console\Application->call("migrate:platform", [--force:true, --pretend:false], Illuminate\Console\OutputStyle)
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:273]
        20. Illuminate\Foundation\Console\Kernel->call("migrate:platform", [--force:true, --pretend:false], Illuminate\Console\OutputStyle)
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php:261]
        21. Illuminate\Support\Facades\Facade::__callStatic("call", ["migrate:platform", [[--force:true, --pretend:false], Illuminate\Console\OutputStyle]])
            [/usr/local/apnscp/lib/Lararia/Console/Commands/Migrate.php:55]
        22. Lararia\Console\Commands\Migrate->handle()
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36]
        23. Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Container/Util.php:37]
        24. Illuminate\Container\Util::unwrapIfClosure()
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93]
        25. Illuminate\Container\BoundMethod::callBoundMethod(Lararia\Application, [], [Lararia\Console\Commands\Migrate, "handle"], )
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37]
        26. Illuminate\Container\BoundMethod::call(Lararia\Application, [], [Lararia\Console\Commands\Migrate, "handle"], null)
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Container/Container.php:590]
        27. Illuminate\Container\Container->call([Lararia\Console\Commands\Migrate, "handle"])
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Console/Command.php:134]
        28. Illuminate\Console\Command->execute(Symfony\Component\Console\Input\ArgvInput, Illuminate\Console\OutputStyle)
            [/usr/local/apnscp/vendor/symfony/console/Command/Command.php:255]
        29. Symfony\Component\Console\Command\Command->run(Symfony\Component\Console\Input\ArgvInput, Illuminate\Console\OutputStyle)
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Console/Command.php:121]
        30. Illuminate\Console\Command->run(Symfony\Component\Console\Input\ArgvInput, Symfony\Component\Console\Output\ConsoleOutput)
            [/usr/local/apnscp/vendor/symfony/console/Application.php:1021]
        31. Symfony\Component\Console\Application->doRunCommand(Lararia\Console\Commands\Migrate, Symfony\Component\Console\Input\ArgvInput, Symfony\Component\Console\Output\ConsoleOutput)
            [/usr/local/apnscp/vendor/symfony/console/Application.php:275]
        32. Symfony\Component\Console\Application->doRun(Symfony\Component\Console\Input\ArgvInput, Symfony\Component\Console\Output\ConsoleOutput)
            [/usr/local/apnscp/vendor/symfony/console/Application.php:149]
        33. Symfony\Component\Console\Application->run(Symfony\Component\Console\Input\ArgvInput, Symfony\Component\Console\Output\ConsoleOutput)
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Console/Application.php:93]
        34. Illuminate\Console\Application->run(Symfony\Component\Console\Input\ArgvInput, Symfony\Component\Console\Output\ConsoleOutput)
            [/usr/local/apnscp/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:131]
        35. Illuminate\Foundation\Console\Kernel->handle(Symfony\Component\Console\Input\ArgvInput, Symfony\Component\Console\Output\ConsoleOutput)
            [/usr/local/apnscp/lib/Lararia/Console/Kernel.php:31]
        36. Lararia\Console\Kernel->handle(Symfony\Component\Console\Input\ArgvInput, Symfony\Component\Console\Output\ConsoleOutput)
            [/usr/local/apnscp/artisan:69]

The entire postfix folder is missing in /home/virtual/FILESYSTEMTEMPLATE/siteinfo//etc/ but just on this machine.

8

Run upcp -sbf to reinitialize the filesystem. That directory must always exist; it’s a requisite component.

9

That command was running for about an hour now and exited with a fail:

TASK [php/common : Query PHP API version] ******************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "cmd": "/home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin/php-config --extension-dir", "msg": "[Errno 2] File or directory not found: b'/home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin/php-config': b'/home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin/php-config'", "rc": 2}
10

More context on that would be useful. Does /.socket/php/usr/bin/php-config exists? If so, does /home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin exist and is it a symlink to /home/virtual/FILESYSTEMTEMPLATE/siteinfo/usr/bin? You can provide the results of:

stat  /.socket/php/usr/bin/php-config /home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin /home/virtual/FILESYSTEMTEMPLATE/siteinfo/usr/bin /home/virtual/FILESYSTEMTEMPLATE/siteinfo/usr/bin/php-config

From everything I see, it looks like you removed /home/virtual/FILESYSTEMTEMPLATE/siteinfo at some point, possibly to cleanup space. This would be consistent with the apiscp-int-* domains that couldn’t fully create during self-check routines.

Never remove files within FILESYSTEMTEMPLATE; these files are hardlinked and will add no additional space outside of an inode reference provided /home and / are on the same drive.

# stat /home/virtual/FILESYSTEMTEMPLATE/siteinfo/etc/login.defs
  File: /home/virtual/FILESYSTEMTEMPLATE/siteinfo/etc/login.defs
  Size: 2501            Blocks: 8          IO Block: 4096   regular file
Device: 803h/2051d      Inode: 67760672    Links: 2
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)

So long as the “links” > 0, that file will remain on the drive even if removed. Moreover, removing those files has disastrous consequences for the vfs layers.

11

Thanks Matt.
I’m pretty sure that I’ve not deleted those files because this server has like 80% free space and used to have only one site hosted. I checked logs and monitoring. Apparently, after something, idk what, was running at night on Nov 9th, the CPU load has been at 100% for 14 hours and then the server crashed. After this, the mess started.

[root@s1 ~]# stat /.socket/php/usr/bin/php-config /home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin /home/virtual/FILESYSTEMTEMPLATE/siteinfo/usr/bin /home/virtual/FILESYSTEMTEMPLATE/siteinfo/usr/bin/php-config
  File: /.socket/php/usr/bin/php-config
  Size: 3998           Blocks: 8          IO Block: 4096   regular file
Device: fd02h/64770d     Inode: 37759296    Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-06 11:11:35.615736589 +0100
Modify: 2024-12-06 11:11:28.717689536 +0100
Change: 2024-12-06 11:11:28.722689570 +0100
Birth: 2024-12-06 11:11:28.717689536 +0100
  File: /home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin
  Size: 84             Blocks: 0          IO Block: 4096   directory
Device: fd02h/64770d     Inode: 17640427    Links: 2
Access: (0555/dr-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-06 10:56:21.196501201 +0100
Modify: 2024-12-06 11:32:55.236466037 +0100
Change: 2024-12-06 11:32:55.236466037 +0100
Birth: 2024-12-05 21:47:00.439734647 +0100
  File: /home/virtual/FILESYSTEMTEMPLATE/siteinfo/usr/bin
  Size: 16384          Blocks: 64         IO Block: 4096   directory
Device: fd02h/64770d     Inode: 14433       Links: 2
Access: (0555/dr-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-05 21:49:29.953765669 +0100
Modify: 2024-12-05 21:49:40.508838454 +0100
Change: 2024-12-05 21:49:40.508838454 +0100
Birth: 2024-12-05 19:13:58.539262279 +0100
  File: /home/virtual/FILESYSTEMTEMPLATE/siteinfo/usr/bin/php-config -> /.socket/php/usr/bin/php-config
  Size: 31             Blocks: 0          IO Block: 4096   symbolic link
Device: fd02h/64770d     Inode: 560021      Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-06 10:45:46.485174172 +0100
Modify: 2024-12-05 21:49:40.087835551 +0100
Change: 2024-12-05 21:49:40.087835551 +0100
Birth: 2024-12-05 21:49:40.087835551 +0100
12

This should be a symlink to usr/bin. What files are located within there? find /home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin

13

Doesn’t look like a symlink to /usr/bin/.

[root@s1 ~]# find /home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin
/home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin
/home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin/mail
/home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin/mailx
/home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin/traceroute
/home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin/traceroute6
/home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin/hostname
14

I was able to reproduce the same layout by removing siteinfo/bin/, then running upcp -sbf apnscp/initialize-filesystem-template, so it does look like this happened previously; see attached.

rm -rf /home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin
ln -rs /home/virtual/FILESYSTEMTEMPLATE/siteinfo/usr/bin /home/virtual/FILESYSTEMTEMPLATE/siteinfo/bin
upcp -sbf apnscp/initialize-filesystem-template apnscp/php-filesystem-template

Then once that runs to make sure everything else is OK, perform a full scrub: upcp -sb

upcp-log.txt (43.4 KB)

15

Thank you. This error is gone but the next one comes up. I guess I can save some of our time when reinstalling this machine. It’s unfortunate but the crash might have caused sever damage.
I don’t really know how it could have happened or how to prevent it again but just one out of a few apiscp instances is effected so it might have other reasons.

TASK [software/scl : Populate .modulespath] ****************************************************************************
failed: [localhost] (item=/home/virtual/FILESYSTEMTEMPLATE/siteinfo) => {"ansible_loop_var": "item", "changed": false, "checksum": "490d27d978182aa901c5859ba129f3c319695cc5", "item": "/home/virtual/FILESYSTEMTEMPLATE/siteinfo", "msg": "Destination directory /home/virtual/FILESYSTEMTEMPLATE/siteinfo/usr/share/Modules/init does not exist"}
16

Set the platform on edge if you’re not already on it:

cpcmd scope:set cp.update-policy edge-major

Then,

upcp -bf software/scl

Behavior is fixed in edge to always repopulate vfs if --force is specified.

17

Thank you. The recent change in edge has fixed this issue. Now there appear to be some (unintended) changes in the pyenv repo. No idea how all these files got changes or messed up.

TASK [/usr/local/apnscp/resources/playbooks/roles/software/pyenv/../pyenv-support : Clone pyenv repository] ************
fatal: [localhost]: FAILED! => {"before": "ee84e89ef1a24ea76c9b12f2379784ebeff79d7b", "changed": false, "msg": "Local modifications exist in repository (force=no)."}
18

pyenv is getting moved to apisnetworks/pyenv for better management. Run upcp again to apply the migration that updates this repo.

19

This solved it, thanks. Now we’re back at randomly generated accounts and errors in the test-account task:

TASK [apnscp/assert-account-works : Remove test account] ***************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["/usr/local/apnscp/bin/DeleteDomain", "apiscp-int-ilfwlltnsqabhdsnngwbpmbk.test"], "delta": "0:00:01.532579", "end": "2024-12-06 21:56:11.729320", "msg": "non-zero return code", "rc": 1, "start": "2024-12-06 21:56:10.196741", "stderr": "ERROR  : unknown(): failed to remove site `site2': fatal(): fatal error retrieving URL: `Curl error: SSL certificate problem: certificate has expired'\n         0. Error_Reporter::add_error(\"failed to remove site `%s': %s\", [\"site2\", \"fatal(): fatal error retrieving URL: `Curl error: SSL certificate problem: certificate has expired'\"])\n            [/usr/local/apnscp/lib/log_wrapper.php:72]\n         1. error(\"failed to remove site `%s': %s\", \"site2\", \"fatal(): fatal error retrieving URL: `Curl error: SSL certificate problem: certificate has expired'\")\n            [/usr/local/apnscp/bin/DeleteDomain:99]", "stderr_lines": ["ERROR  : unknown(): failed to remove site `site2': fatal(): fatal error retrieving URL: `Curl error: SSL certificate problem: certificate has expired'", "         0. Error_Reporter::add_error(\"failed to remove site `%s': %s\", [\"site2\", \"fatal(): fatal error retrieving URL: `Curl error: SSL certificate problem: certificate has expired'\"])", "            [/usr/local/apnscp/lib/log_wrapper.php:72]", "         1. error(\"failed to remove site `%s': %s\", \"site2\", \"fatal(): fatal error retrieving URL: `Curl error: SSL certificate problem: certificate has expired'\")", "            [/usr/local/apnscp/bin/DeleteDomain:99]"], "stdout": "", "stdout_lines": []}

New accounts are created but not by me. I think that has something to do with the failing task:

[root@s1 ~]# ls -la /home/virtual/
drwx--x--x. 5 root root 65536  6. Dez 21:56 .
drwxr-xr-x. 3 root root    21 13. Jan 2023  ..
lrwxrwxrwx  1 root root    23  6. Dez 21:56 admin2 -> /home/virtual/site2/fst
lrwxrwxrwx  1 root root    23  6. Dez 21:56 apiscp-int-ilfwlltnsqabhdsnngwbpmbk.test -> /home/virtual/site2/fst
drwxr-xr-x  5 root root    60  5. Dez 19:13 FILESYSTEMTEMPLATE
drwx--x--x  5 root root    43  5. Dez 19:12 site1
drwx--x--x  5 root root    43  6. Dez 21:56 site2
20 
AddDomain  -c siteinfo,domain=daskjldfsdfs.com -c siteinfo,admin_user=sdfkldfs -c dns,enabled=0
env DEBUG=1 DeleteDomain daskjldfsdfs.com

That should give you a bit more insight… Possibly running cp-proxy with an expired certificate?