Database error after update with Update Assurance enabled

Support
1

Bug Report Template

Description

Webapp Update failed on a WordPress site. Update Assurance was enabled. After the attempt and presumably a rollback, site was inaccessible with the error, “Error establishing a database connection."

I eventually identified the problem as about 3/4 of the tables missing from the database. I restored from a database backup and was able to resolve the issue.

Expected Behavior

Update Assurance should restore the site and database to the state prior to the update when there is an error.

Actual Behavior

The database became corrupted.

Environment

ApisCP version:

revision: b36a59c8fd7253d363abc4917e5f400cc2e3daf1
timestamp: 1725962694
ver_maj: 3
ver_min: 2
ver_patch: 45
ver_pre: ''
dirty: false
debug: false

Operating System: AlmaLinux 8 4.18.0-553.16.1.el8_10.x86_64

Additional relevant information (e.g., PHP version, database, etc.): database is MariaDB

2

Database export is stored in .db-dump.sql within the app root. I would first examine that to ensure all tables are properly there, as well as reapplying the export there.

3

Comparing the file size .db-dump.sql from last night’s automatic update to my backup that I know to be good, as well as a previously versioned copy of the file, shows them all to be roughly the same size. No obvious problems. (Incidentally, the file I successfully restored from was one generated and stored in the site owner’s home directory mysql_backups.)

This database is on the larger side, with the dump consisting of about 650MB. Could that be causing an issue?

4

Is cgroup,memory set on the site? This could be a problem.

Database backup would be analogous to cpcmd -d DOMAIN mysql:export DBNAME /var/www/html/.db-dump.sql. This part checks out.

Importing first empties, then imports. The command sequence would be:

cpcmd -d DOMAIN mysql:empty-db DBNAME
cpcmd -d DOMAIN mysql:import /var/www/html/.db-dump.sql

Restoring from the UI is the same process albeit, API signatures are outdated wrappers. Did you restore from within the user’s terminal or UI, which would be governed by cgroup restrictions or as root?

5

cgroup,memory is set to 1024 MB for the site.

When I successfully restored from a backup, I believe I was acting as root and executed mysql < db-dump.sql.

So, the automatic process works through the user account? I could imagine the cgroup limit getting in the way. I would think 1024MB would be enough memory, but maybe not.

6

Try running the command tandem, let’s see if it reproduces the problem:

cpcmd -d DOMAIN mysql:empty-db DBNAME
cpcmd -d DOMAIN mysql:import /var/www/html/.db-dump.sql

You can verify an OOM condition was reached by inspecting dmesg output or checking /sys/fs/cgroup/<SITE>/memory.events.

7

I am unable to reproduce the problem with this test. This is a live site, so I didn’t want to break it, but I made a copy of the database and tried these commands. I see in memory.events that there have been OOM conditions in the past, but no new ones when I export or import the database. During my testing, I didn’t see the memory usage go over 40%.

As a side note, cpcmd -d DOMAIN mysq:empty-db DBNAME failed with ERROR: apnscpFunctionInterceptor::check_permissions(): command ``mysql_empty_db' does not exist. Did you mean mysql_empty’?`. Instead, I used the UI as the user to empty the database.