Add maintenance:repair step for file size of unencrypted files

[simonspa]

• Related: Nextcloud displays wrong used storage #25283

Summary

This resets the unencrypted_size of files with encryption = 0 to 0. As described in the linked issue at length, there seem to be setups that either have or had encryption enabled in the past, where even for unencrypted files the unencrypted_size column shows non-zero values.

These seem to completely confuse the quota calculation and result in partially unusable systems because users might have a huge quota usage reported while the actual used size is just a few MB.

TODO

• Add tests

Checklist

• Code is properly formatted
• Sign-off message is added to all commits
• Tests (unit, integration, api and/or acceptance) are included
• Screenshots before/after for front-end changes
• Documentation (manuals or wiki) has been updated or is not required
• Backports requested where applicable (ex: critical bugfixes)

[PVince81]

👍 from me, I hope the second reviewer knows more about DB perf

good luck appeasing the bots 😄
thanks a lot!

[szaimen]

Drone failures look related

[juliusknorr]

There is no index on neither of those columns, so we may want to add at least to the unencrypted size.

I think the most safe way to use such an index then would be:

Suggested change

	->where($update->expr()->eq('encrypted', "0", IQueryBuilder::PARAM_INT))
	->andWhere($update->expr()->neq('unencrypted_size', "0", IQueryBuilder::PARAM_INT));
	->andWhere($update->expr()->gt('unencrypted_size', "0", IQueryBuilder::PARAM_INT));
	->andWhere($update->expr()->eq('encrypted', "0", IQueryBuilder::PARAM_INT))

[juliusknorr]

For the reasoning:

• neq may not use the index
• the order of where statements sometimes also matters to decide if the index is applied

[juliusknorr]

EXPLAIN of the current query does a full table scan:

explain update oc_filecache set unencrypted_size = 0 where encrypted = 0 and unencrypted_size != 0;
+------+-------------+--------------+-------+---------------+---------+---------+------+------+-------------+
| id   | select_type | table        | type  | possible_keys | key     | key_len | ref  | rows | Extra       |
+------+-------------+--------------+-------+---------------+---------+---------+------+------+-------------+
|    1 | SIMPLE      | oc_filecache | index | NULL          | PRIMARY | 8       | NULL | 8479 | Using where |
+------+-------------+--------------+-------+---------------+---------+---------+------+------+-------------+

[simonspa]

After reversal of arguments and using > query affects smaller number of rows:

explain update oc_filecache set unencrypted_size = 0 where unencrypted_size > 0 and encrypted = 0;
+------+-------------+--------------+-------+---------------+---------+---------+------+------+-------------+
| id   | select_type | table        | type  | possible_keys | key     | key_len | ref  | rows | Extra       |
+------+-------------+--------------+-------+---------------+---------+---------+------+------+-------------+
|    1 | SIMPLE      | oc_filecache | index | NULL          | PRIMARY | 8       | NULL | 490  | Using where |
+------+-------------+--------------+-------+---------------+---------+---------+------+------+-------------+

[simonspa]

I implemented the suggested change - but adding an index to either of these columns would have to done separately, and probably run manually, right? How would that work together with an automatic repair step?

[simonspa]

Performance tests is failing with

fatal: remote error: upload-pack: not our ref 03c3817ff132653c794fd04410977952f69fd614

• is this owing to the fact that this pull comes from my fork rather than a branch within this repo?

[sorbaugh]

@simonspa this PR looks nice :) I'm reopening in the hopes that you're still interested in pushing this forward! Since you're a member of Nextcloud, could you recreate this PR in a branch directly in the nextcoud/server repo? Feel free and encouraged to check the Nextcloud Commit guidelines. Hope we can keep this cool improvement alive! 🚀