4G file size limit , in 5+ config (still missing one?) driving me nuts

Support
1

When uploading a file bigger than 4G, it gets stuck at 4G or 4294967296 bytes. Obviously this is the same good-old 4G file size limit we have here in ApisCP. This time however I’m unable to solve it. I know there’s SO MANY config locations for this 4G limit but these are the ones I’ve changed so far. Please see command output below.

I need assistance in what config I’ve missed changing to allow the httpd web server / php to store a file bigger than 4G. I’ve attempted to change this to 20G without success

Thank you for the assistance!

systemctl edit httpd
LimitFSIZE=20G

systemctl show -pLimitFSIZE httpd
LimitFSIZE=21474836480

systemctl edit php-fpm-site3.service
LimitFSize=20G

systemctl show -pLimitFSIZE php-fpm-site3.service
LimitFSIZE=infinity

cat /home/virtual/site3/info/php-policy.yml
resources: { file_size: 20G, oom_adjust: 500 }

cat apnscp-vars-runtime.yml
limit_fsize_soft: 20971520

ulimit_vfs_custom_config:
  fsize: { soft: 20971520 }


su site3
ulimit -f
20971520
2

What’s the HTTP response code? What application is accepting the upload? Does it always fail at the 4 GB boundary?

You’d need to alter the value in:

  • limits.conf using system.process-limits, this affects the max file size created within vfs; can be tested within the account using dd if=/dev/zero of=~/file bs=1G count=5
  • Apache may be overrode using LimitFSIZE, its primary purpose is to throttle logfiles from running away
  • If uploaded by from a PHP app, per-site by overriding resources.file_size

Lastly, PHP mounts a tmpfs mount for /tmp. If uploads under 4 GB but more than 1 GB are failing, then this is a consequence of using a private /tmp mount. It may be adjusted in the policy file.

Resource limits in php-fpm.service or php-fpm-siteXX.service do not propagate to the individual pools which control the physical startup of PHP-FPM. These services are named php-fpm-siteXX-POOL-NAME.service. Policy overrides will propagate to these services however.

3

The HTTP response code becomes 503

The method is POST upload, php application.
The total file size is 10G.

It always fail at exactly 4G / 4294967296 bytes. When viewing the terminal I can see that it’s storing the file in the uploads folder under /var/www and stops at the above limit.

Doing: dd if=/dev/zero of=~/file bs=1G count=5 works fine.

Understand I’ve added
[Service]
LimitFSize=20G

to the php-fpm-siteXX-POOL-NAME.service now as well.

For the apache, all looks good?

systemctl edit httpd.service
[Service]
MemoryAccounting=true
MemoryMax=4G
Delegate=yes
LimitFSIZE=20G
4

Looks fine to me. Here’s a sample test:

<form method="post" enctype="multipart/form-data">
<input type="file" name='x' />
<input type="submit" />
</form>

<pre><code>
<?php var_dump($_FILES);

Create a 1 GB + 5 GB sample:

dd if=/dev/zero of=/test-sm bs=1G count=1
dd if=/dev/zero of=/test-lg bs=1G count=5

Override .user.ini configuration in document root:

upload_max_filesize=6G
post_max_size=6G

Disable PrivateTmp, which is memory-backed given the file size requirement. Also increase the LimitFSIZE value that’s set when the PHP-FPM process spins up:

cpcmd php:pool-set-policy site2 privatetmp false
cpcmd php:pool-set-policy site2 resources.file_size 6G

Then simulate the request:

curl --resolve abc.test:80:172.18.47.186 http://abc.test/test.php -F "x=@/test-lg"

<form method="post" enctype="multipart/form-data">
<input type="file" name='x' />
<input type="submit" />
</form>

<pre><code>
array(1) {
  ["x"]=>
  array(6) {
    ["name"]=>
    string(4) "test-lg"
    ["full_path"]=>
    string(4) "test-lg"
    ["type"]=>
    string(24) "application/octet-stream"
    ["tmp_name"]=>
    string(14) "/tmp/phpcowPhB"
    ["error"]=>
    int(0)
    ["size"]=>
    int(5368709120)
  }
}
1 Like