Bug Report Template
Description
Webapp Update failed on a WordPress site. Update Assurance was enabled. After the attempt and presumably a rollback, site was inaccessible with the error, “Error establishing a database connection."
I eventually identified the problem as about 3/4 of the tables missing from the database. I restored from a database backup and was able to resolve the issue.
Expected Behavior
Update Assurance should restore the site and database to the state prior to the update when there is an error.
Actual Behavior
The database became corrupted.
Environment
ApisCP version:
revision: b36a59c8fd7253d363abc4917e5f400cc2e3daf1
timestamp: 1725962694
ver_maj: 3
ver_min: 2
ver_patch: 45
ver_pre: ''
dirty: false
debug: false
Operating System: AlmaLinux 8 4.18.0-553.16.1.el8_10.x86_64
Additional relevant information (e.g., PHP version, database, etc.): database is MariaDB
Database export is stored in .db-dump.sql within the app root. I would first examine that to ensure all tables are properly there, as well as reapplying the export there.
Comparing the file size .db-dump.sql
from last night’s automatic update to my backup that I know to be good, as well as a previously versioned copy of the file, shows them all to be roughly the same size. No obvious problems. (Incidentally, the file I successfully restored from was one generated and stored in the site owner’s home directory mysql_backups.)
This database is on the larger side, with the dump consisting of about 650MB. Could that be causing an issue?
Is cgroup,memory set on the site? This could be a problem.
Database backup would be analogous to cpcmd -d DOMAIN mysql:export DBNAME /var/www/html/.db-dump.sql
. This part checks out.
Importing first empties, then imports. The command sequence would be:
cpcmd -d DOMAIN mysql:empty-db DBNAME
cpcmd -d DOMAIN mysql:import /var/www/html/.db-dump.sql
Restoring from the UI is the same process albeit, API signatures are outdated wrappers. Did you restore from within the user’s terminal or UI, which would be governed by cgroup restrictions or as root?
cgroup,memory is set to 1024 MB for the site.
When I successfully restored from a backup, I believe I was acting as root and executed mysql < db-dump.sql
.
So, the automatic process works through the user account? I could imagine the cgroup limit getting in the way. I would think 1024MB would be enough memory, but maybe not.
Try running the command tandem, let’s see if it reproduces the problem:
cpcmd -d DOMAIN mysql:empty-db DBNAME
cpcmd -d DOMAIN mysql:import /var/www/html/.db-dump.sql
You can verify an OOM condition was reached by inspecting dmesg
output or checking /sys/fs/cgroup/<SITE>/memory.events.
I am unable to reproduce the problem with this test. This is a live site, so I didn’t want to break it, but I made a copy of the database and tried these commands. I see in memory.events that there have been OOM conditions in the past, but no new ones when I export or import the database. During my testing, I didn’t see the memory usage go over 40%.
As a side note, cpcmd -d DOMAIN mysq:empty-db DBNAME
failed with ERROR: apnscpFunctionInterceptor::check_permissions(): command ``mysql_empty_db' does not exist. Did you mean
mysql_empty’?`. Instead, I used the UI as the user to empty the database.