transfersite failures

Support
1

Trying to migrate two CentOS 7 servers to two new Rocky 8 servers.
Both new servers setup identically and one of them is failing when transferring sites.

This is the full output and I’ve never seen an issue like this.
The servers are on Edge Major

beginning migration: customerdomain.com (stage 0)
creating site on p100.lithium.hosting
  + different source/dest platform 8.0/8.0
  + unsetting ssh port index
synchronizing users
copying user config
synchronizing passwords
syncing mysql databases...
creating `siteuser_dbuser1'
adding backup task for siteuser_dbuser1
adding siteuser@localhost
adding mysql grant for `siteuser'@`localhost' on `siteuser_dbuser1'
adding siteuser@127.0.0.1
syncing mysql db `siteuser_dbuser1'
LOCAL: rsync -HW /home/virtual/site1/fst/tmp/siteuser_dbuser1.sql root@p100.lithium.hosting:/home/virtual/site1/fst/tmp/siteuser_dbuser1.sql
syncing pgsql databases...
syncing postgresql users...
adding siteuser
syncing subdomains
syncing multiphp...
LOCAL: rsync -aHWx /home/virtual/site1/info/php-policy.yml root@p100.lithium.hosting:/home/virtual/site1/info/php-policy.yml
syncing transports...
syncing mailing lists...
syncing files...
LOCAL: find /home/virtual/site1/shadow -ignore_readdir_race -nowarn -xdev \( -not -group root \( \( -path '/home/virtual/site1/shadow/home/*' -o -path '/home/virtual/site1/shadow/home' -o -path '/home/virtual/site1/shadow/var/www/*' -o -path '/home/virtual/site1/shadow/var/www' -o -path '/home/virtual/site1/shadow/usr/local/*' -o -path '/home/virtual/site1/shadow/usr/local' -o -path '/home/virtual/site1/shadow/etc/localtime/*' -o -path '/home/virtual/site1/shadow/etc/localtime' -o -path '/home/virtual/site1/shadow/etc/usertemplate/*' -o -path '/home/virtual/site1/shadow/etc/usertemplate' -o -path '/home/virtual/site1/shadow/etc/logrotate.d/*' -o -path '/home/virtual/site1/shadow/etc/logrotate.d' -o -path '/home/virtual/site1/shadow/etc/maildroprc/*' -o -path '/home/virtual/site1/shadow/etc/maildroprc' -o -path '/home/virtual/site1/shadow/var/lib/majordomo/*' -o -path '/home/virtual/site1/shadow/var/lib/majordomo' -o -path '/home/virtual/site1/shadow/etc/mail/spamassassin/local.cf/*' -o -path '/home/virtual/site1/shadow/etc/mail/spamassassin/local.cf' -o -path '/home/virtual/site1/shadow/etc/vsftpd/*' -o -path '/home/virtual/site1/shadow/etc/vsftpd' \) \( -type f -o -type l -o -type d -a -empty \) \) \) -fprintf /home/virtual/site1/shadow/filelist.txt "/%P\0" || true
LOCAL: rsync --stats -aHWxzA --delete --delete-excluded --usermap=9993:9999,989:tomcat --groupmap=10001:1003,989:tomcat --numeric-ids --files-from=/home/virtual/site1/shadow/filelist.txt -0 /home/virtual/site1/shadow root@p100.lithium.hosting:/home/virtual/site1/fst
filesystem xfer finished: 
Number of files: 486 (reg: 404, dir: 81, link: 1)
Number of created files: 436 (reg: 394, dir: 42)
Number of deleted files: 0
Number of regular files transferred: 404
Total file size: 9,310,522 bytes
Total transferred file size: 9,310,504 bytes
Literal data: 9,310,504 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.087 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 5,007,138
Total bytes received: 7,929

sent 5,007,138 bytes  received 7,929 bytes  3,343,378.00 bytes/sec
total size is 9,310,522  speedup is 1.86

syncing mailboxes
skipping vmount sync
NOTICE: copying custom http config
LOCAL: rsync -aHWx /etc/httpd/conf/site1.ssl/custom root@p100.lithium.hosting:/etc/httpd/conf/site1.ssl/custom
syncing dns...
syncing ssl...
LOCAL: rsync -aHWx /usr/local/apnscp/storage/certificates/data/certs/acme-v02.api.letsencrypt.org.directory/site1/ root@p100.lithium.hosting:/usr/local/apnscp/storage/certificates/data/certs/acme-v02.api.letsencrypt.org.directory/site1
updating dns stage data
INTERNAL REPORT: TypeError::__set_state(array(
   'message' => 'preg_replace(): Argument #3 ($subject) must be of type array|string, null given',
   'string' => '',
   'code' => 0,
   'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',
   'line' => 82,
   'trace' => 
  array (
    0 => 
    array (
      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',
      'line' => 82,
      'function' => 'preg_replace',
      'args' => 
      array (
        0 => '/^((?:(?:"(?>[^"]+)(?>"\\s*)))*)(?!")(.+)$/',
        1 => '\\1"\\2"',
        2 => NULL,
      ),
    ),
    1 => 
    array (
      'file' => '/usr/local/apnscp/lib/Opcenter/Dns/Record.php',
      'line' => 67,
      'function' => 'formatTxt',
      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Record',
      'type' => '->',
      'args' => 
      array (
      ),
    ),
    2 => 
    array (
      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',
      'line' => 23,
      'function' => '__construct',
      'class' => 'Opcenter\\Dns\\Record',
      'type' => '->',
      'args' => 
      array (
        0 => 'customerdomain.com',
        1 => 
        array (
          'name' => '__acct_migration',
          'rr' => 'txt',
          'parameter' => NULL,
        ),
      ),
    ),
    3 => 
    array (
      'file' => '/usr/local/apnscp/lib/Module/Support/Dns.php',
      'line' => 155,
      'function' => '__construct',
      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Record',
      'type' => '->',
      'args' => 
      array (
        0 => 'customerdomain.com',
        1 => 
        array (
          'name' => '__acct_migration',
          'rr' => 'txt',
          'parameter' => NULL,
        ),
      ),
    ),
    4 => 
    array (
      'file' => '/usr/local/apnscp/lib/modules/dns.php',
      'line' => 1318,
      'function' => 'createRecord',
      'class' => 'Module\\Support\\Dns',
      'type' => '::',
      'args' => 
      array (
        0 => 'customerdomain.com',
        1 => 
        array (
          'name' => '__acct_migration',
          'rr' => 'txt',
          'parameter' => NULL,
        ),
      ),
    ),
    5 => 
    array (
      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Module.php',
      'line' => 351,
      'function' => 'record_exists',
      'class' => 'Dns_Module',
      'type' => '->',
      'args' => 
      array (
        0 => 'customerdomain.com',
        1 => '__acct_migration',
        2 => 'txt',
        3 => '',
      ),
    ),
    6 => 
    array (
      'file' => '/usr/local/apnscp/lib/Module/Skeleton/Standard.php',
      'line' => 145,
      'function' => 'record_exists',
      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Module',
      'type' => '->',
      'args' => 
      array (
        0 => 'customerdomain.com',
        1 => '__acct_migration',
        2 => 'txt',
      ),
    ),
    7 => 
    array (
      'file' => '/usr/local/apnscp/lib/apnscpfunction.php',
      'line' => 992,
      'function' => '_invoke',
      'class' => 'Module\\Skeleton\\Standard',
      'type' => '->',
      'args' => 
      array (
        0 => 'record_exists',
        1 => 
        array (
          0 => 'customerdomain.com',
          1 => '__acct_migration',
          2 => 'txt',
        ),
      ),
    ),
    8 => 
    array (
      'file' => '/usr/local/apnscp/lib/apnscpfunction.php',
      'line' => 932,
      'function' => 'call',
      'class' => 'apnscpFunctionInterceptor',
      'type' => '->',
      'args' => 
      array (
        0 => 'dns_record_exists',
        1 => 
        array (
          0 => 'customerdomain.com',
          1 => '__acct_migration',
          2 => 'txt',
        ),
      ),
    ),
    9 => 
    array (
      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',
      'line' => 101,
      'function' => '__call',
      'class' => 'apnscpFunctionInterceptor',
      'type' => '->',
      'args' => 
      array (
        0 => 'dns_record_exists',
        1 => 
        array (
          0 => 'customerdomain.com',
          1 => '__acct_migration',
          2 => 'txt',
        ),
      ),
    ),
    10 => 
    array (
      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',
      'line' => 1865,
      'function' => '__call',
      'class' => 'CLI_Transfer',
      'type' => '->',
      'args' => 
      array (
        0 => 'dns_record_exists',
        1 => 
        array (
          0 => 'customerdomain.com',
          1 => '__acct_migration',
          2 => 'txt',
        ),
      ),
    ),
    11 => 
    array (
      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',
      'line' => 1828,
      'function' => 'set_migration_stage',
      'class' => 'CLI_Transfer',
      'type' => '->',
      'args' => 
      array (
        0 => 'customerdomain.com',
        1 => 1,
      ),
    ),
    12 => 
    array (
      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',
      'line' => 200,
      'function' => '_sync_update_stage',
      'class' => 'CLI_Transfer',
      'type' => '->',
      'args' => 
      array (
      ),
    ),
    13 => 
    array (
      'file' => '/usr/local/apnscp/bin/scripts/transfersite.php',
      'line' => 233,
      'function' => 'process',
      'class' => 'CLI_Transfer',
      'type' => '->',
      'args' => 
      array (
      ),
    ),
  ),
   'previous' => NULL,
)) 

         0B. Error_Reporter::handle_error(32, "TypeError::__set_state(array(   'message' => 'preg_replace(): Argument #3 ($subject) must be of type array|string, null given',   'string' => '',   'code' => 0,   'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',   'line' => 82,   'trace' =>   array (    0 =>     array (      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',      'line' => 82,      'function' => 'preg_replace',      'args' =>       array (        0 => '/^((?:(?:"(?>[^"]+)(?>"\\s*)))*)(?!")(.+)$/',        1 => '\\1"\\2"',        2 => NULL,      ),    ),    1 =>     array (      'file' => '/usr/local/apnscp/lib/Opcenter/Dns/Record.php',      'line' => 67,      'function' => 'formatTxt',      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Record',      'type' => '->',      'args' =>       array (      ),    ),    2 =>     array (      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',      'line' => 23,      'function' => '__construct',      'class' => 'Opcenter\\Dns\\Record',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 =>         array (          'name' => '__acct_migration',          'rr' => 'txt',          'parameter' => NULL,        ),      ),    ),    3 =>     array (      'file' => '/usr/local/apnscp/lib/Module/Support/Dns.php',      'line' => 155,      'function' => '__construct',      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Record',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 =>         array (          'name' => '__acct_migration',          'rr' => 'txt',          'parameter' => NULL,        ),      ),    ),    4 =>     array (      'file' => '/usr/local/apnscp/lib/modules/dns.php',      'line' => 1318,      'function' => 'createRecord',      'class' => 'Module\\Support\\Dns',      'type' => '::',      'args' =>       array (        0 => 'customerdomain.com',        1 =>         array (          'name' => '__acct_migration',          'rr' => 'txt',          'parameter' => NULL,        ),      ),    ),    5 =>     array (      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Module.php',      'line' => 351,      'function' => 'record_exists',      'class' => 'Dns_Module',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 => '__acct_migration',        2 => 'txt',        3 => '',      ),    ),    6 =>     array (      'file' => '/usr/local/apnscp/lib/Module/Skeleton/Standard.php',      'line' => 145,      'function' => 'record_exists',      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Module',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 => '__acct_migration',        2 => 'txt',      ),    ),    7 =>     array (      'file' => '/usr/local/apnscp/lib/apnscpfunction.php',      'line' => 992,      'function' => '_invoke',      'class' => 'Module\\Skeleton\\Standard',      'type' => '->',      'args' =>       array (        0 => 'record_exists',        1 =>         array (          0 => 'customerdomain.com',          1 => '__acct_migration',          2 => 'txt',        ),      ),    ),    8 =>     array (      'file' => '/usr/local/apnscp/lib/apnscpfunction.php',      'line' => 932,      'function' => 'call',      'class' => 'apnscpFunctionInterceptor',      'type' => '->',      'args' =>       array (        0 => 'dns_record_exists',        1 =>         array (          0 => 'customerdomain.com',          1 => '__acct_migration',          2 => 'txt',        ),      ),    ),    9 =>     array (      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',      'line' => 101,      'function' => '__call',      'class' => 'apnscpFunctionInterceptor',      'type' => '->',      'args' =>       array (        0 => 'dns_record_exists',        1 =>         array (          0 => 'customerdomain.com',          1 => '__acct_migration',          2 => 'txt',        ),      ),    ),    10 =>     array (      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',      'line' => 1865,      'function' => '__call',      'class' => 'CLI_Transfer',      'type' => '->',      'args' =>       array (        0 => 'dns_record_exists',        1 =>         array (          0 => 'customerdomain.com',          1 => '__acct_migration',          2 => 'txt',        ),      ),    ),    11 =>     array (      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',      'line' => 1828,      'function' => 'set_migration_stage',      'class' => 'CLI_Transfer',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 => 1,      ),    ),    12 =>     array (      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',      'line' => 200,      'function' => '_sync_update_stage',      'class' => 'CLI_Transfer',      'type' => '->',      'args' =>       array (      ),    ),    13 =>     array (      'file' => '/usr/local/apnscp/bin/scripts/transfersite.php',      'line' => 233,      'function' => 'process',      'class' => 'CLI_Transfer',      'type' => '->',      'args' =>       array (      ),    ),  ),   'previous' => NULL,))", null, null, Exception)
            [/usr/local/apnscp/lib/error_reporter.php:1344]
         1B. Error_Reporter::report("TypeError::__set_state(array(   'message' => 'preg_replace(): Argument #3 ($subject) must be of type array|string, null given',   'string' => '',   'code' => 0,   'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',   'line' => 82,   'trace' =>   array (    0 =>     array (      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',      'line' => 82,      'function' => 'preg_replace',      'args' =>       array (        0 => '/^((?:(?:"(?>[^"]+)(?>"\\s*)))*)(?!")(.+)$/',        1 => '\\1"\\2"',        2 => NULL,      ),    ),    1 =>     array (      'file' => '/usr/local/apnscp/lib/Opcenter/Dns/Record.php',      'line' => 67,      'function' => 'formatTxt',      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Record',      'type' => '->',      'args' =>       array (      ),    ),    2 =>     array (      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',      'line' => 23,      'function' => '__construct',      'class' => 'Opcenter\\Dns\\Record',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 =>         array (          'name' => '__acct_migration',          'rr' => 'txt',          'parameter' => NULL,        ),      ),    ),    3 =>     array (      'file' => '/usr/local/apnscp/lib/Module/Support/Dns.php',      'line' => 155,      'function' => '__construct',      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Record',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 =>         array (          'name' => '__acct_migration',          'rr' => 'txt',          'parameter' => NULL,        ),      ),    ),    4 =>     array (      'file' => '/usr/local/apnscp/lib/modules/dns.php',      'line' => 1318,      'function' => 'createRecord',      'class' => 'Module\\Support\\Dns',      'type' => '::',      'args' =>       array (        0 => 'customerdomain.com',        1 =>         array (          'name' => '__acct_migration',          'rr' => 'txt',          'parameter' => NULL,        ),      ),    ),    5 =>     array (      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Module.php',      'line' => 351,      'function' => 'record_exists',      'class' => 'Dns_Module',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 => '__acct_migration',        2 => 'txt',        3 => '',      ),    ),    6 =>     array (      'file' => '/usr/local/apnscp/lib/Module/Skeleton/Standard.php',      'line' => 145,      'function' => 'record_exists',      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Module',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 => '__acct_migration',        2 => 'txt',      ),    ),    7 =>     array (      'file' => '/usr/local/apnscp/lib/apnscpfunction.php',      'line' => 992,      'function' => '_invoke',      'class' => 'Module\\Skeleton\\Standard',      'type' => '->',      'args' =>       array (        0 => 'record_exists',        1 =>         array (          0 => 'customerdomain.com',          1 => '__acct_migration',          2 => 'txt',        ),      ),    ),    8 =>     array (      'file' => '/usr/local/apnscp/lib/apnscpfunction.php',      'line' => 932,      'function' => 'call',      'class' => 'apnscpFunctionInterceptor',      'type' => '->',      'args' =>       array (        0 => 'dns_record_exists',        1 =>         array (          0 => 'customerdomain.com',          1 => '__acct_migration',          2 => 'txt',        ),      ),    ),    9 =>     array (      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',      'line' => 101,      'function' => '__call',      'class' => 'apnscpFunctionInterceptor',      'type' => '->',      'args' =>       array (        0 => 'dns_record_exists',        1 =>         array (          0 => 'customerdomain.com',          1 => '__acct_migration',          2 => 'txt',        ),      ),    ),    10 =>     array (      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',      'line' => 1865,      'function' => '__call',      'class' => 'CLI_Transfer',      'type' => '->',      'args' =>       array (        0 => 'dns_record_exists',        1 =>         array (          0 => 'customerdomain.com',          1 => '__acct_migration',          2 => 'txt',        ),      ),    ),    11 =>     array (      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',      'line' => 1828,      'function' => 'set_migration_stage',      'class' => 'CLI_Transfer',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 => 1,      ),    ),    12 =>     array (      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',      'line' => 200,      'function' => '_sync_update_stage',      'class' => 'CLI_Transfer',      'type' => '->',      'args' =>       array (      ),    ),    13 =>     array (      'file' => '/usr/local/apnscp/bin/scripts/transfersite.php',      'line' => 233,      'function' => 'process',      'class' => 'CLI_Transfer',      'type' => '->',      'args' =>       array (      ),    ),  ),   'previous' => NULL,))", ["TypeError::__set_state(array(   'message' => 'preg_replace(): Argument #3 ($subject) must be of type array|string, null given',   'string' => '',   'code' => 0,   'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',   'line' => 82,   'trace' =>   array (    0 =>     array (      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',      'line' => 82,      'function' => 'preg_replace',      'args' =>       array (        0 => '/^((?:(?:"(?>[^"]+)(?>"\\s*)))*)(?!")(.+)$/',        1 => '\\1"\\2"',        2 => NULL,      ),    ),    1 =>     array (      'file' => '/usr/local/apnscp/lib/Opcenter/Dns/Record.php',      'line' => 67,      'function' => 'formatTxt',      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Record',      'type' => '->',      'args' =>       array (      ),    ),    2 =>     array (      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',      'line' => 23,      'function' => '__construct',      'class' => 'Opcenter\\Dns\\Record',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 =>         array (          'name' => '__acct_migration',          'rr' => 'txt',          'parameter' => NULL,        ),      ),    ),    3 =>     array (      'file' => '/usr/local/apnscp/lib/Module/Support/Dns.php',      'line' => 155,      'function' => '__construct',      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Record',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 =>         array (          'name' => '__acct_migration',          'rr' => 'txt',          'parameter' => NULL,        ),      ),    ),    4 =>     array (      'file' => '/usr/local/apnscp/lib/modules/dns.php',      'line' => 1318,      'function' => 'createRecord',      'class' => 'Module\\Support\\Dns',      'type' => '::',      'args' =>       array (        0 => 'customerdomain.com',        1 =>         array (          'name' => '__acct_migration',          'rr' => 'txt',          'parameter' => NULL,        ),      ),    ),    5 =>     array (      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Module.php',      'line' => 351,      'function' => 'record_exists',      'class' => 'Dns_Module',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 => '__acct_migration',        2 => 'txt',        3 => '',      ),    ),    6 =>     array (      'file' => '/usr/local/apnscp/lib/Module/Skeleton/Standard.php',      'line' => 145,      'function' => 'record_exists',      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Module',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 => '__acct_migration',        2 => 'txt',      ),    ),    7 =>     array (      'file' => '/usr/local/apnscp/lib/apnscpfunction.php',      'line' => 992,      'function' => '_invoke',      'class' => 'Module\\Skeleton\\Standard',      'type' => '->',      'args' =>       array (        0 => 'record_exists',        1 =>         array (          0 => 'customerdomain.com',          1 => '__acct_migration',          2 => 'txt',        ),      ),    ),    8 =>     array (      'file' => '/usr/local/apnscp/lib/apnscpfunction.php',      'line' => 932,      'function' => 'call',      'class' => 'apnscpFunctionInterceptor',      'type' => '->',      'args' =>       array (        0 => 'dns_record_exists',        1 =>         array (          0 => 'customerdomain.com',          1 => '__acct_migration',          2 => 'txt',        ),      ),    ),    9 =>     array (      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',      'line' => 101,      'function' => '__call',      'class' => 'apnscpFunctionInterceptor',      'type' => '->',      'args' =>       array (        0 => 'dns_record_exists',        1 =>         array (          0 => 'customerdomain.com',          1 => '__acct_migration',          2 => 'txt',        ),      ),    ),    10 =>     array (      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',      'line' => 1865,      'function' => '__call',      'class' => 'CLI_Transfer',      'type' => '->',      'args' =>       array (        0 => 'dns_record_exists',        1 =>         array (          0 => 'customerdomain.com',          1 => '__acct_migration',          2 => 'txt',        ),      ),    ),    11 =>     array (      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',      'line' => 1828,      'function' => 'set_migration_stage',      'class' => 'CLI_Transfer',      'type' => '->',      'args' =>       array (        0 => 'customerdomain.com',        1 => 1,      ),    ),    12 =>     array (      'file' => '/usr/local/apnscp/lib/CLI/Transfer.php',      'line' => 200,      'function' => '_sync_update_stage',      'class' => 'CLI_Transfer',      'type' => '->',      'args' =>       array (      ),    ),    13 =>     array (      'file' => '/usr/local/apnscp/bin/scripts/transfersite.php',      'line' => 233,      'function' => 'process',      'class' => 'CLI_Transfer',      'type' => '->',      'args' =>       array (      ),    ),  ),   'previous' => NULL,))"])
            [/usr/local/apnscp/lib/apnscpcore.php:145]
         2B. Error_Reporter\ReportFilterInterface@anonymous```
2

I’ve tried multiple accounts with the same issue, tried resetting DNS to default with no success.

3

Updated support for weak record matching in PowerDNS module on edge.

1 Like
4

Ok that worked for DNS but now subdomains fail. This used to work flawlessly, why so many issues with transfers lately?

syncing subdomains
NOTICE: web_add_subdomain failed... retrying...
NOTICE: web_add_subdomain failed... retrying...
NOTICE: web_add_subdomain failed... retrying...
NOTICE: web_add_subdomain failed... retrying...
NOTICE: web_add_subdomain failed... retrying...
FAILURE: web_add_subdomain
fatal(): `web_add_subdomain': crash or other nasty error detected 
Line 71 
#0 /usr/local/apnscp/lib/Util/API.php(71): SoapClient->__call('web_add_subdoma...', Array)
#1 /usr/local/apnscp/lib/CLI/Transfer.php(243): Util_API->__call('web_add_subdoma...', Array)
#2 /usr/local/apnscp/lib/CLI/Transfer.php(1103): CLI_Transfer->_sync('remote', 'web_add_subdoma...', 'cheatingbulldog...', '/var/www/html/c...')
#3 /usr/local/apnscp/lib/CLI/Transfer.php(173): CLI_Transfer->_sync_subdomains()
#4 /usr/local/apnscp/bin/scripts/transfersite.php(233): CLI_Transfer->process()
#5 {main}
ERROR  : unknown(): migration failed - unhandled exception
         0B. Error_Reporter::add_error("migration failed - unhandled exception", )
            [/usr/local/apnscp/lib/log_wrapper.php:72]
         1B. error("migration failed - unhandled exception")
            [/usr/local/apnscp/bin/scripts/transfersite.php:247]
unknown(): migration failed - unhandled exception

It does seem to affect multiple accounts on both servers I’m running migrations on. If I delete the subdomains, the transfer succeeds. I thought maybe it was related to subdomains in /home/user/ but it’s also in /var/www.

5

I’d need to see the backend log in storage/logs/start.log on the migration destination to provide further guidance.

6

Ask and you shall receive. Different account, same error on the source server with this on the destination. edge-major rocky 8

[Sat Jul 27 02:21:55 2024] Added customerdomain1.com (WARNING) Succeeded
[Sat Jul 27 02:21:57 2024] Edited site8, customerdomain1.com (SUCCESS) Succeeded
[Sat Jul 27 02:21:58 2024] Edited site8, customerdomain1.com (SUCCESS) Succeeded
INFO   : created database `evel_db1'
INFO   : created database `evel_db2'
[Sat Jul 27 02:22:08 2024] Edited site8, customerdomain1.com (WARNING) Succeeded
WARNING: Util_Account_Editor::_parseOutput(): Dns_Module::remove_zone(): Bypassing DNS removal. DNS UUID for `customerdomain2.com' is `94506564130237215083e7408214-b326-509d-8eaa-b8fa398b0be6'. Server UUID is `34108932519398701766355dde263-e507-586b-ba68-627392bb118c'
WARNING: Util_Account_Editor::_parseOutput(): Dns_Module::remove_zone(): Bypassing DNS removal. DNS UUID for `customerdomain3.com' is `94506564130237215083e7408214-b326-509d-8eaa-b8fa398b0be6'. Server UUID is `34108932519398701766355dde263-e507-586b-ba68-627392bb118c'
INFO   : Hang tight! Domain changes will be active within a few minutes, but may take up to 24 hours to work properly.
[Sat Jul 27 02:22:08 2024] [last message repeated 383 times]
[Sat Jul 27 02:22:10 2024] Edited site8, customerdomain1.com (WARNING) Succeeded
WARNING: Util_Account_Editor::_parseOutput(): Dns_Module::add_zone(): DNS for zone `customerdomain2.com' already exists, not overwriting
WARNING: Util_Account_Editor::_parseOutput(): Dns_Module::add_zone(): DNS for zone `customerdomain3.com' already exists, not overwriting
INFO   : Hang tight! Domain changes will be active within a few minutes, but may take up to 24 hours to work properly.
INFO   : Record brautigan.customerdomain1.com already exists - not overwriting
[Sat Jul 27 02:22:11 2024] INTERNAL REPORT: TypeError::__set_state(array(
   'message' => 'rtrim(): Argument #1 ($string) must be of type string, null given',
   'string' => '',
   'code' => 0,
   'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',
   'line' => 105,
   'trace' => 
  array (
    0 => 
    array (
      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',
      'line' => 105,
      'function' => 'rtrim',
      'args' => 
      array (
        0 => NULL,
        1 => '.',
      ),
    ),
    1 => 
    array (
      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',
      'line' => 28,
      'function' => 'appendDot',
      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Record',
      'type' => '->',
      'args' => 
      array (
      ),
    ),
    2 => 
    array (
      'file' => '/usr/local/apnscp/lib/Opcenter/Dns/Record.php',
      'line' => 67,
      'function' => 'formatCname',
      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Record',
      'type' => '->',
      'args' => 
      array (
      ),
    ),
    3 => 
    array (
      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Record.php',
      'line' => 23,
      'function' => '__construct',
      'class' => 'Opcenter\\Dns\\Record',
      'type' => '->',
      'args' => 
      array (
        0 => 'customerdomain1.com',
        1 => 
        array (
          'name' => 'www.brautigan',
          'rr' => 'CNAME',
          'parameter' => NULL,
        ),
      ),
    ),
    4 => 
    array (
      'file' => '/usr/local/apnscp/lib/Module/Support/Dns.php',
      'line' => 155,
      'function' => '__construct',
      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Record',
      'type' => '->',
      'args' => 
      array (
        0 => 'customerdomain1.com',
        1 => 
        array (
          'name' => 'www.brautigan',
          'rr' => 'CNAME',
          'parameter' => NULL,
        ),
      ),
    ),
    5 => 
    array (
      'file' => '/usr/local/apnscp/lib/modules/dns.php',
      'line' => 1318,
      'function' => 'createRecord',
      'class' => 'Module\\Support\\Dns',
      'type' => '::',
      'args' => 
      array (
        0 => 'customerdomain1.com',
        1 => 
        array (
          'name' => 'www.brautigan',
          'rr' => 'CNAME',
          'parameter' => NULL,
        ),
      ),
    ),
    6 => 
    array (
      'file' => '/usr/local/apnscp/resources/playbooks/roles/software/powerdns-support/src/Module.php',
      'line' => 351,
      'function' => 'record_exists',
      'class' => 'Dns_Module',
      'type' => '->',
      'args' => 
      array (
        0 => 'customerdomain1.com',
        1 => 'www.brautigan',
        2 => 'CNAME',
        3 => '',
      ),
    ),
    7 => 
    array (
      'file' => '/usr/local/apnscp/lib/modules/dns.php',
      'line' => 1122,
      'function' => 'record_exists',
      'class' => 'Opcenter\\Dns\\Providers\\Powerdns\\Module',
      'type' => '->',
      'args' => 
      array (
        0 => 'customerdomain1.com',
        1 => 'www.brautigan',
        2 => 'CNAME',
      ),
    ),
    8 => 
    array (
      'file' => '/usr/local/apnscp/lib/Module/Skeleton/Standard.php',
      'line' => 145,
      'function' => 'add_record_conditionally',
      'class' => 'Dns_Module',
      'type' => '->',
      'args' => 
      array (
        0 => 'customerdomain1.com',
        1 => 'www.brautigan',
        2 => 'A',
        3 => '167.253.62.1',
      ),
    ),
    9 => 
    array (
      'file' => '/usr/local/apnscp/lib/apnscpfunction.php',
      'line' => 992,
      'function' => '_invoke',
      'class' => 'Module\\Skeleton\\Standard',
      'type' => '->',
      'args' => 
      array (
        0 => 'add_record_conditionally',
        1 => 
        array (
          0 => 'customerdomain1.com',
          1 => 'www.brautigan',
          2 => 'A',
          3 => '167.253.62.1',
        ),
      ),
    ),
    10 => 
    array (
      'file' => '/usr/local/apnscp/lib/apnscpFunctionInterceptorTrait.php',
      'line' => 34,
      'function' => 'call',
      'class' => 'apnscpFunctionInterceptor',
      'type' => '->',
      'args' => 
      array (
        0 => 'dns_add_record_conditionally',
        1 => 
        array (
          0 => 'customerdomain1.com',
          1 => 'www.brautigan',
          2 => 'A',
          3 => '167.253.62.1',
        ),
      ),
    ),
    11 => 
    array (
      'file' => '/usr/local/apnscp/lib/modules/web.php',
      'line' => 1346,
      'function' => '__call',
      'class' => 'Module\\Skeleton\\Standard',
      'type' => '->',
      'args' => 
      array (
        0 => 'dns_add_record_conditionally',
        1 => 
        array (
          0 => 'customerdomain1.com',
          1 => 'www.brautigan',
          2 => 'A',
          3 => '167.253.62.1',
        ),
      ),
    ),
    12 => 
    array (
      'file' => '/usr/local/apnscp/lib/Module/Skeleton/Standard.php',
      'line' => 145,
      'function' => 'add_subdomain',
      'class' => 'Web_Module',
      'type' => '->',
      'args' => 
      array (
        0 => 'brautigan.customerdomain1.com',
        1 => '/var/www/brautigan.customerdomain1.com',
      ),
    ),
    13 => 
    array (
      'file' => '/usr/local/apnscp/lib/apnscpfunction.php',
      'line' => 992,
      'function' => '_invoke',
      'class' => 'Module\\Skeleton\\Standard',
      'type' => '->',
      'args' => 
      array (
        0 => 'add_subdomain',
        1 => 
        array (
          0 => 'brautigan.customerdomain1.com',
          1 => '/var/www/brautigan.customerdomain1.com',
        ),
      ),
    ),
    14 => 
    array (
      'file' => '/usr/local/apnscp/lib/lservicelib8.pht',
      'line' => 1185,
      'function' => 'call',
      'class' => 'apnscpFunctionInterceptor',
      'type' => '->',
      'args' => 
      array (
        0 => 'web_add_subdomain',
        1 => 
        array (
          0 => 'brautigan.customerdomain1.com',
          1 => '/var/www/brautigan.customerdomain1.com',
        ),
      ),
    ),
    15 => 
    array (
      'file' => '/usr/local/apnscp/lib/lservicelib8.pht',
      'line' => 1102,
      'function' => 'process_backend_data',
      'class' => 'ListenerService\\Daemon',
      'type' => '->',
      'args' => 
      array (
        0 => '�9' . "\0" . 'O:12:"apnscpObject":8:{s:7:"command";s:17:"web_add_subdomain";s:4:"args";a:2:{i:0;s:25:"brautigan.customerdomain1.com";i:1;s:34:"/var/www/brautigan.customerdomain1.com";}s:5:"error";N;s:10:"session_id";s:32:"XXXXX";s:13:"returned_data";N;s:7:"options";N;s:12:"error_buffer";N;s:16:"' . "\0" . 'apnscpObject' . "\0" . 'ip";s:13:"54.39.154.240";}' . "\0" . '' . "\0" . '' . "\0" . '',
      ),
    ),
    16 => 
    array (
      'file' => '/usr/local/apnscp/lib/lservicelib8.pht',
      'line' => 1058,
      'function' => 'client_processing_loop',
      'class' => 'ListenerService\\Daemon',
      'type' => '->',
      'args' => 
      array (
      ),
    ),
    17 => 
    array (
      'file' => '/usr/local/apnscp/lib/lservicelib8.pht',
      'line' => 866,
      'function' => 'spawn',
      'class' => 'ListenerService\\Daemon',
      'type' => '->',
      'args' => 
      array (
      ),
    ),
    18 => 
    array (
      'file' => '/usr/local/apnscp/lib/lservicelib8.pht',
      'line' => 1411,
      'function' => 'findWorker',
      'class' => 'ListenerService\\Daemon',
      'type' => '->',
      'args' => 
      array (
        0 => 0,
      ),
    ),
    19 => 
    array (
      'file' => '/usr/local/apnscp/lib/lservicelib8.pht',
      'line' => 1283,
      'function' => 'dispatch',
      'class' => 'ListenerService\\Daemon',
      'type' => '->',
      'args' => 
      array (
        0 => 
        Socket::__set_state(array(
        )),
      ),
    ),
    20 => 
    array (
      'file' => '/usr/local/apnscp/lib/lservicelib8.pht',
      'line' => 1231,
      'function' => 'master',
      'class' => 'ListenerService\\Daemon',
      'type' => '->',
      'args' => 
      array (
      ),
    ),
    21 => 
    array (
      'file' => '/usr/local/apnscp/lib/lservicelib8.pht',
      'line' => 293,
      'function' => 'create_master_socket_server',
      'class' => 'ListenerService\\Daemon',
      'type' => '->',
      'args' => 
      array (
      ),
    ),
    22 => 
    array (
      'file' => '/usr/local/apnscp/lib/lservicelib8.pht',
      'line' => 171,
      'function' => 'start',
      'class' => 'ListenerService\\Daemon',
      'type' => '->',
      'args' => 
      array (
      ),
    ),
    23 => 
    array (
      'file' => '/usr/local/apnscp/lib/lservicelib8.pht',
      'line' => 1464,
      'function' => '__construct',
      'class' => 'ListenerService\\Daemon',
      'type' => '->',
      'args' => 
      array (
      ),
    ),
    24 => 
    array (
      'file' => '/usr/local/apnscp/lib/lservice.php',
      'line' => 28,
      'function' => 'init',
      'class' => 'ListenerService\\Daemon',
      'type' => '::',
      'args' => 
      array (
      ),
    ),
  ),
   'previous' => NULL,
)) 


         0B. Error_Reporter::get_debug_bt()
            [/usr/local/apnscp/lib/error_reporter.php:281]
         1B. Error_Reporter::handle_error(32, "", null, null, Exception)
            [/usr/local/apnscp/lib/error_reporter.php:1344]
         2B. Error_Reporter::report("", [""])
            [/usr/local/apnscp/lib/apnscpcore.php:145]
         3B. Error_Reporter\ReportFilterInterface@anonymous
7

Should be fixed now. All DNS modules are running local record caches as a workaround for APIs which don’t synchronously publish the record changes. This introduced problems with weak matches (fetching records without a parameter).

I’m missing areas in current test coverage that stresses this feature.

8

Anything I can do to help while migrating hundreds of accounts?

9

TTL isn’t being changed on stage 0, only for the TXT record for __acct_migration which is 5.

lh-t9c6
lh-t9c61132×742 33.6 KB

10

upcp, resolved :+1:

1 Like
11

Any chance there’s a way to update those records for migrations that have been submitted at stage 0 already? We’re talking a few hundred accounts.

12

Use change_dns.php

./bin/scripts/change_dns.php --ttl=15 --all

It’s also possible to re-run initial migration without creation by adding --no-create --stage=0 to the migration script.

13

Thanks, the only other thing I see is lots of failures due to duplicate rrsets and it’s almost always roundcube…

Reason: Failed to update record 'roundcube' on zone 'domain.com' (old - rr: 'A', param: '54.39.163.98'; new - name: 'roundcube' rr: 'A', param: '54.39.163.98'): Duplicate RRset roundcube.domain.com. IN A with changetype: REPLACE
14

I decided to use EXT4 on these new servers but that’s posing a new issue.

ERROR  : Util_Process::formatDataCallProc(): xfs_quota: xfs_quota: cannot find project DEFAULT

         0B. Error_Reporter::add_error("xfs_quota: xfs_quota: cannot find project DEFAULT", )
            [/usr/local/apnscp/lib/log_wrapper.php:72]
         1B. error("xfs_quota: xfs_quota: cannot find project DEFAULT")
            [/usr/local/apnscp/lib/Util/Process.php:1383]
         2B. Util_Process->formatDataCallProc()
            [/usr/local/apnscp/lib/Util/Process.php:1350]
         3B. Util_Process->format("apnscp")
            [/usr/local/apnscp/lib/Util/Process.php:800]
         4B. Util_Process->_run()
            [/usr/local/apnscp/lib/Util/Process.php:467]
         5B. Util_Process->run(["/usr/sbin/xfs_quota", "-x", "-c", "quota -pNbi %s", "/"], ["DEFAULT"])
            [/usr/local/apnscp/lib/Util/Process.php:295]
         6B. Util_Process::exec(["/usr/sbin/xfs_quota", "-x", "-c", "quota -pNbi %s", "/"], ["DEFAULT"])
            [/usr/local/apnscp/lib/Opcenter/Filesystem/Quota/Project.php:241]
         7B. Opcenter\Filesystem\Quota\Project->exec("quota -pNbi %s", ["DEFAULT"])
            [/usr/local/apnscp/lib/Opcenter/Filesystem/Quota/Project.php:138]
         8B. Opcenter\Filesystem\Quota\Project->get()
            [/usr/local/apnscp/lib/modules/site.php:220]
         9B. Site_Module->get_account_quota()
            [/usr/local/apnscp/lib/Module/Skeleton/Standard.php:145]
        10B. Module\Skeleton\Standard->_invoke("get_account_quota", )
            [/usr/local/apnscp/lib/apnscpfunction.php:992]
        11B. apnscpFunctionInterceptor->call("site_get_account_quota", )
            [/usr/local/apnscp/lib/apnscpFunctionInterceptorTrait.php:34]
        12B. Module\Skeleton\Standard->__call("site_get_account_quota", )
            [/usr/local/apnscp/lib/modules/mysql.php:2259]
15

Got it. Service definition is missing the new value, which inherits a dependent default value that requires transformation. EditDomain --all on the target server after upcp. Ignore any errors it generates with respect to missing project files.

Strongly discouraged option; it hasn’t been the default filesystem since RHEL6. Using ext4 locks you out from using reseller/project quota feature in ApisCP. Moreover, from what I saw on an NVMe with fio benchmarks on Rocky 8, it resulted in lower throughput with higher CPU utilization.

16

Yep, understood. It was an oversight on my part with a VPS template, it’s been fixed for future deployments.

Another issue came up, when an account is transferred, the source server is sending password change notices and customers are getting scared and confused. Can these be suppressed during a transfer?

17

New Issue… The value was empty, it was never ZERO.

writeiops=None
readiops=None
cpupin=None
delegator=None

Reason: fatal(): `aliases_add_domain': crash or other nasty error detected
array (
  0 => 
  array (
    'message' => 'Util_Account_Editor::_parseOutput(): Opcenter\\Service\\Validators\\Cgroup\\Readbw::valid(): Converting value to null. cgroup,writeiops value of `0\' can cause system lockups.',
    'severity' => 8,
    'caller' => 'Util_Account_Editor::_parseOutput',
    'bt' => NULL,
  ),
  1 => 
  array (
    'message' => 'Util_Account_Editor::_parseOutput(): Opcenter\\Service\\Validators\\Cgroup\\Readbw::valid(): Converting value to null. cgroup,readiops value of `0\' can cause system lockups.',
    'severity' => 8,
    'caller' => 'Util_Account_Editor::_parseOutput',
    'bt' => NULL,
  ),
  2 => 
  array (
    'message' => 'Hang tight! Domain changes will be active within a few minutes, but may take up to 24 hours to work properly.',
    'severity' => 4,
    'caller' => 'Aliases_Module::_synchronize_changes',
    'bt' => NULL,
  ),
)
18

This roundcube record issue is happening on almost every single transfer.
On Stage 0, the system is creating a new roundcube record with the new server IP, then trying to update the TTL to 5 and fails because there are two A records, one for the new and one for the old IP.

Seems to be mail, horde, and roundcube records that get duplicated.

Reason: Failed to update record 'roundcube' on zone 'customerdomain.com' (old - rr: 'A', param: '54.39.163.98'; new - name: 'roundcube' rr: 'A', param: '54.39.163.98'): Duplicate RRset roundcube.customerdomain.com. IN A with changetype: REPLACE
         0B. SoapClient->__call("dns_modify_record", ["customerdomain.com", "roundcube", "A", "54.39.163.98", [[ttl:5, parameter:"54.39.163.98"]]])
            [/usr/local/apnscp/lib/Util/API.php:71]
         1B. Util_API->__call("dns_modify_record", ["customerdomain.com", "roundcube", "A", "54.39.163.98", [[ttl:5, parameter:"54.39.163.98"]]])
            [/usr/local/apnscp/lib/CLI/Transfer.php:243]
         2B. CLI_Transfer->_sync("remote", "dns_modify_record", "customerdomain.com", "roundcube", "A", "54.39.163.98", [ttl:5, parameter:"54.39.163.98"])
            [/usr/local/apnscp/lib/CLI/Transfer.php:258]
         3B. CLI_Transfer->_sync("remote", "dns_modify_record", "customerdomain.com", "roundcube", "A", "54.39.163.98", [ttl:5, parameter:"54.39.163.98"])
            [/usr/local/apnscp/lib/CLI/Transfer.php:258]
         4B. CLI_Transfer->_sync("remote", "dns_modify_record", "customerdomain.com", "roundcube", "A", "54.39.163.98", [ttl:5, parameter:"54.39.163.98"])
            [/usr/local/apnscp/lib/CLI/Transfer.php:258]
         5B. CLI_Transfer->_sync("remote", "dns_modify_record", "customerdomain.com", "roundcube", "A", "54.39.163.98", [ttl:5, parameter:"54.39.163.98"])
            [/usr/local/apnscp/lib/CLI/Transfer.php:258]
         6B. CLI_Transfer->_sync("remote", "dns_modify_record", "customerdomain.com", "roundcube", "A", "54.39.163.98", [ttl:5, parameter:"54.39.163.98"])
            [/usr/local/apnscp/lib/CLI/Transfer.php:258]
         7B. CLI_Transfer->_sync("remote", "dns_modify_record", "customerdomain.com", "roundcube", "A", "54.39.163.98", [ttl:5, parameter:"54.39.163.98"])
            [/usr/local/apnscp/lib/CLI/Transfer.php:1902]
         8B. CLI_Transfer->alter_ip("customerdomain.com", "54.39.163.98", "54.39.163.98", 5)
            [/usr/local/apnscp/lib/CLI/Transfer.php:1837]
         9B. CLI_Transfer->_sync_update_stage()
            [/usr/local/apnscp/lib/CLI/Transfer.php:200]
        10B. CLI_Transfer->process()
            [/usr/local/apnscp/bin/scripts/transfersite.php:233]


array (
)

I just tried retransferring a dozen accounts on stage 0 and they all failed because of this.

19

More addon domain issues? Since when does migrating require valid nameservers?

syncing addon domains...
Adding domain customerdomain1.com -> /var/www/html
NOTICE: aliases_add_domain failed... retrying...
NOTICE: aliases_add_domain failed... retrying...
NOTICE: aliases_add_domain failed... retrying...
NOTICE: aliases_add_domain failed... retrying...
NOTICE: aliases_add_domain failed... retrying...
FAILURE: aliases_add_domain
Aliases_Module::_verify_url(): Verification URL request moved to different location other than accepted: http://www.customerdomain1.com/e77883299449cdfd31e10c127d62ef185c031417.html
Aliases_Module::_verify(): `customerdomain1.com': domain has DNS records delegated to nameservers dns1.registrar-servers.com, dns2.registrar-servers.com. Domain cannot be added to this account for security. Complete one of the following options to verify ownership:

(1) Change nameservers to ns1.lithiumdns.net, ns2.lithiumdns.net within the domain registrar
(2) Upload a html file to your old hosting provider accessible via http://customerdomain1.com/e77883299449cdfd31e10c127d62ef185c031417.html with the content:
        e77883299449cdfd31e10c127d62ef185c031417
(3) Create a temporary DNS record named newacct.customerdomain1.com with an `A' resource record that points to 167.253.62.1

Please contact your previous hosting provider for assistance with performing any of these verification options. 
Line 71 
#0 /usr/local/apnscp/lib/Util/API.php(71): SoapClient->__call('aliases_add_dom...', Array)
#1 /usr/local/apnscp/lib/CLI/Transfer.php(243): Util_API->__call('aliases_add_dom...', Array)
#2 /usr/local/apnscp/lib/CLI/Transfer.php(1080): CLI_Transfer->_sync('remote', 'aliases_add_dom...', 'customerdomain1.com', '/var/www/html')
#3 /usr/local/apnscp/lib/CLI/Transfer.php(172): CLI_Transfer->_sync_addon_domains()
#4 /usr/local/apnscp/bin/scripts/transfersite.php(233): CLI_Transfer->process()
#5 {main}
FIRST != LAST: fatal(): `aliases_add_domain': crash or other nasty error detected 
Line 71 
#0 /usr/local/apnscp/lib/Util/API.php(71): SoapClient->__call('aliases_add_dom...', Array)
#1 /usr/local/apnscp/lib/CLI/Transfer.php(243): Util_API->__call('aliases_add_dom...', Array)
#2 /usr/local/apnscp/lib/CLI/Transfer.php(258): CLI_Transfer->_sync('remote', 'aliases_add_dom...', 'customerdomain1.com', '/var/www/html')
#3 /usr/local/apnscp/lib/CLI/Transfer.php(258): CLI_Transfer->_sync('remote', 'aliases_add_dom...', 'customerdomain1.com', '/var/www/html')
#4 /usr/local/apnscp/lib/CLI/Transfer.php(258): CLI_Transfer->_sync('remote', 'aliases_add_dom...', 'customerdomain1.com', '/var/www/html')
#5 /usr/local/apnscp/lib/CLI/Transfer.php(258): CLI_Transfer->_sync('remote', 'aliases_add_dom...', 'customerdomain1.com', '/var/www/html')
#6 /usr/local/apnscp/lib/CLI/Transfer.php(258): CLI_Transfer->_sync('remote', 'aliases_add_dom...', 'customerdomain1.com', '/var/www/html')
#7 /usr/local/apnscp/lib/CLI/Transfer.php(1080): CLI_Transfer->_sync('remote', 'aliases_add_dom...', 'customerdomain1.com', '/var/www/html')
#8 /usr/local/apnscp/lib/CLI/Transfer.php(172): CLI_Transfer->_sync_addon_domains()
#9 /usr/local/apnscp/bin/scripts/transfersite.php(233): CLI_Transfer->process()
#10 {main}
ERROR  : unknown(): migration failed - unhandled exception
         0B. Error_Reporter::add_error("migration failed - unhandled exception", )
            [/usr/local/apnscp/lib/log_wrapper.php:72]
         1B. error("migration failed - unhandled exception")
            [/usr/local/apnscp/bin/scripts/transfersite.php:247]
unknown(): migration failed - unhandled exception
20

Password change emails are suppressed when auth,pwoverride=1 is set on the acount. This is done during password sync. Likewise I am unable to reproduce this with migrating a test account to another server, e.g.

AddDomain  -c siteinfo,domain=testtest.dev -c siteinfo,admin_user=sfdkjdf -c siteinfo,email=matt@apisnetworks.com -c dns,enabled=1
./bin/scripts/transfersite.php -s falcon testtest.dev
./bin/scripts/transfersite.php -s falcon --force testtest.dev

Are you able to reliably reproduce this?

About 17 years ago. This server has domain validation enabled under [domains] => dns_check and nameservers configured for the account don’t match the nameservers configured under [dns] => hosting_ns. Migrations use the API, which is indistinguishable from user interaction, so the same security measures are in place to prevent adding a domain a customer doesn’t have access to.

In a multi-server setup, cp-proxy will consult your master database to ensure the domain has the same billing,invoice service value. If this checks, the domain may be added. Configure [auth] settings on the new server so the dns:domain-on-account check succeeds. cpcmd -d DOMAIN dns:domain-on-account ADDON-DOMAIN should report 1 on the target server if cp-proxy is setup properly.

Can you reliably reproduce this with:

cpcmd -d customerdomain.com dns:modify-record customerdomain.com roundcube A 54.39.163.98 '[ttl:5, parameter:54.39.163.98]'

If so, what does this report?

cpcmd -d customerdomain.com dns:get-records 'roundcube' A customerdomain.com