Learning Mode cleanup exceeds ARG_MAX after 12,243 changed paths

Bug reports
, , ,
1

Description

Matomo’s updater was initially unable to write application files because ApisCP Fortification was active. I selected Learning Mode under Web > Web Apps, updated Matomo from 5.10.0 to 5.14.1 during the learning window, and later received a File Change Report for 12,243 changed paths.

When Learning Mode finished, ACL enforcement failed with:

setfacl: proc_open(): Exec failed: Argument list too long

Enforcement results (unlock changed files):
    Partially succeeded.

Environment

  • ApisCP: v3.2.48-89-g2547ff9e5 (revision 2547ff9e59908d364183572a2d2b55ec297505ee)
  • OS: AlmaLinux 8.10
  • ARG_MAX: 2,097,152 bytes
  • Web application: Matomo
  • Application update: 5.10.0 to 5.14.1
  • Changed paths reported: 12,243
  • Fortification action: Learning Mode
  • Enforcement mode used by the scheduled job: unlock

Current upstream master appears to retain the same command construction.

Steps that triggered the problem

  1. Attempt a Matomo update while Fortification prevents the updater from writing application files.
  2. Open Web > Web Apps and select Learning Mode for the Matomo installation.
  3. Run the Matomo 5.10.0 to 5.14.1 update during the learning window.
  4. Allow the scheduled Learning Mode job to finish.
  5. Receive a File Change Report containing 12,243 changed paths and the Argument list too long error.

I have not repeated the update on a disposable account. These steps describe the observed production incident.

Learning Mode execution sequence

Learning Mode initially grants recursive ACLs by passing one application-root path to file_set_acls(), then creates a checkpoint and schedules watch_batch after the learning duration:

The initial grant supplies one root path and lets setfacl -R perform the traversal. The large argument list is created later.

After the learning window, watch_batch compares the checkpoint with the current tree and passes the diff to Watch_Module::lockdown():

In the unlock branch, lockdown() first changes ownership and removes ACLs recursively using the single application-root path. It then expands the changed files and calls:

$ret = $this->file_set_acls($filteredFiles, $users);

Changed directories are processed only if that call succeeds:

File_Module::_acl_driver() creates one placeholder for every supplied path and passes the entire path array to a single process:

$cmd = 'setfacl ' . $flags . ' ' . join(' ', $rights);
$cmd .= str_repeat(' %s', count($files));
$proc = Util_Process_Safe::exec($cmd, $files);

Diagnosis

The 12,243 explicit path arguments, together with their path lengths and the process environment, appear to exceed the operating system’s execve() argument-size limit. The proc_open(): Exec failed: Argument list too long error, the measured ARG_MAX, and the current _acl_driver() implementation support this diagnosis.

Current impact

The Matomo site remained available. After the failure I verified:

  • no named ACL entries remain in the Matomo tree;
  • no application files or directories outside Matomo’s runtime tmp directory are group- or world-writable;
  • the PHP-FPM user can read the application entry point and write to tmp;
  • public application requests return HTTP 200.

There is no evidence of compromise or unauthorized modification. The report nevertheless says enforcement only partially succeeded.

Expected behavior

Learning Mode cleanup should complete for a valid change set even when the changed paths do not fit in one process argument vector.

A possible fix would be to split $files into byte-aware batches before calling Util_Process_Safe::exec(). A path-count limit alone could still exceed ARG_MAX when paths are long. Each batch should preserve the flags and ACL rights, and the operation should report the exact failed batch.

Recovery questions

What is the supported way to complete Learning Mode cleanup after this partial failure?

  1. Can the original diff safely be passed to watch:lockdown(..., 'unlock') again?
  2. Can watch:batch safely be repeated with the original checkpoint?
  3. Should Fortification be reapplied through Web > Web Apps instead?
  4. Does the current execution order mean the recursive ACL removal completes before the changed-file ACL grant fails?
  5. Is there a supported command that verifies whether Learning Mode left any changed paths with incomplete ACLs?

I can provide a redacted copy of the report, path-length statistics, and additional diagnostics if useful.

2

Learning mode isn’t intended for one-off in-place updates. Learning mode is intended to provide the fewest number of changes to allow an untrusted user (PHP-FPM pool owner) write-access to your application without knowing what access is required. The reason 12k files changed ownership is because Matomo performed a bulk update of all files. All of these files would be writeable by PHP, which defeats the purpose of privilege separation afforded by Fortification.

It’s appropriate to either change ownership of these files to “apache” under Web > Web Apps > Options > Owner or whatever the pool owner is under Web > PHP Pools or place the application in write mode under Fortification. If I were updating Matomo, I’d do so as the account owner from the terminal to ensure an attacker cannot write to protected system files.

See also Fortification | ApisCP Docs

3

I’ve added support for chunk processing, limited to 1024 files at a time. ARG_MAX is max string size, so theoretically 2^21 chars ÷ PATH_MAX (4096) + NUL as positional separator ~ 511 max arguments.

It’s still not an advisable use of Learning Mode, but at least it won’t fall on its face if used in such a manner. :+1:

4

Thank you for the extensive responses @msaladna !

Indeed my thinking was off. My line of thinking was that since Matomo frequently has updates, Learning Mode would easily allow those to take place in the future.

I will follow your advice and implement a cronjob to do the upgrades automatically, also saves me time :slight_smile: