Add occ command to expire the trashbin by DeepDiver1975 · Pull Request #25878 · owncloud/core

DeepDiver1975 · 2016-08-19T15:39:27Z

mention-bot · 2016-08-19T15:39:28Z

@DeepDiver1975, thanks for your PR! By analyzing the annotation information on this pull request, we identified @schiessle and @LukasReschke to be potential reviewers

mmattel · 2016-08-19T15:51:58Z

Great !
Beside expiring the trashbin for a/all users, I would additionally propose the ability to see what and how much data will be deleted. With the current process I just say do it.
Therefore I propose:

--dryrun to see what would be deleted and how much data would be freed - without doing it
--verbose to see online what is deleted
in the results for the user how much actually has been deleted (number and quantity)
a summary when finished deleting x files for y users freeing z MB

PVince81 · 2016-08-22T06:42:27Z

apps/files_trashbin/lib/Command/ExpireTrash.php

+		$maxAge = $this->expiration->getMaxAgeAsTimestamp();
+		if (!$maxAge) {
+			$output->writeln("No expiry configured.");
+//			return;


stray return ?

PVince81 · 2016-08-22T06:46:03Z

@DeepDiver1975 what's there to discuss ? I heard the primary reason for adding this would be to be able to run OCC commands from cron to ease the load from cron ?

PVince81 · 2016-08-22T08:15:05Z

CC @cdamken

cdamken · 2016-08-22T10:50:05Z

Awesome!! Thanks Guys!

@DeepDiver1975 I think this https://github.com/owncloud/enterprise/issues/1279 is would be fixed with this PR too

cdamken · 2016-08-22T10:53:46Z

@carlaschroder This should be added in the documentation ;)

DeepDiver1975 · 2016-08-22T10:54:24Z

@carlaschroder This should be added in the documentation ;)

fore sure - once we agree on this ...

cdamken · 2016-08-22T10:55:21Z

@DeepDiver1975 how about the versions?, could be added too?

DeepDiver1975 · 2016-08-22T11:10:29Z

how about the versions?, could be added too?

on it ....

cdamken · 2016-08-22T11:11:44Z

on it ....

I own you some beers ;)

DeepDiver1975 · 2016-08-22T11:41:23Z

@mmattel thanks for your input - we can enhance this over time in follow up prs.

mmattel · 2016-08-22T11:50:28Z

Ok to me,
pls consider that we use a different way defining users in eg files:scan (like I proposed above) than we do here. imho this should be consistant. would make future improvments also be consistant to other commands...

PVince81 · 2016-08-22T15:55:22Z

apps/files_trashbin/lib/Command/ExpireTrash.php

+
+		$maxAge = $this->expiration->getMaxAgeAsTimestamp();
+		if (!$maxAge) {
+			$output->writeln("No expiry configured.");


I'm getting this now. I thought we had defaults ? There was a thing about expiring trashbin items whenever the trashbin size reached 50% of the total available size

Maybe it's new behavior, I can see that this command simply reflects the same behavior as the background job, so it's ok. Would be good to get some input from @VicDeo

As far as I understood the code this whole expiry is purly based on time ... but I maybe I missed something in this mess .....

@PVince81 as per #16300 that I documented in https://github.com/owncloud/core/blob/master/config/config.sample.php#L415 a value of
auto guarantees neither keeping nor deletion. Items might be deleted if a disk usage is over 50% of quota.
While exact values specify number of days to store / to delete after for min/max respectively.
From that point auto value means the mode before retention obligation was implemented. It is limited by quota usage for trashbin and by by number of copies per different time ranges for versions

Anyway the best description was provided by @MTRichards in #16300 for Trashbin and #16301 for Versions.
This is actually a reference spec for retention

PVince81 · 2016-08-22T16:01:00Z

I tried setting "trashbin_retention_obligation" to "auto" but it still says it's unconfigured. A quick debug in the "Expiration" code shows that it keeps the max age to "no obligation". So it looks like the background job wouldn't expire the trashbin based on size even though the doc says it would when set to "auto".
Anyway, this is not related to this PR.

PVince81 · 2016-08-22T16:03:51Z

ok, looks like the size-based behavior is another single-shot expire job that gets scheduled only after a deletion.

PVince81 · 2016-08-22T16:05:21Z

This is the other expire job for size:

| 14 | OC\Command\CommandJob                             | "O:33:\"OCA\\Files_Trashbin\\Command\\Expire\":1:{s:39:\"\u0000OCA\\Files_Trashbin\\Command\\Expire\u0000user\";s:36:\"80cd743a-fcb2-1035-90e4-53fe065c9f7b\";}" |          0 |   1471881096 |           0 |
| 15 | OC\Command\CommandJob                             | "O:33:\"OCA\\Files_Trashbin\\Command\\Expire\":1:{s:39:\"\u0000OCA\\Files_Trashbin\\Command\\Expire\u0000user\";s:36:\"80cde582-fcb2-1035-90e5-53fe065c9f7b\";}" |          0 |   1471881193 |           0 |

Looks like it would be too tricky to add this behavior into this command as well.

Ok ok, I'll now test with numeric values.

PVince81 · 2016-08-22T16:24:33Z

Tested, works 👍

(it was really painful to manipulate timestamps in oc_filecache as I forgot to also re-set the path_hash since the timestamp is also in the filename)

PVince81 · 2016-08-22T16:24:33Z

Tested, works 👍

(it was really painful to manipulate timestamps in oc_filecache as I forgot to also re-set the path_hash since the timestamp is also in the filename)

PVince81 · 2016-08-22T16:39:49Z

Ah, that was the trashbin stuff.

Now testing versions, let me put the steps now:

Setup OC with Mysql or MariaDB
Set "versions_retention_obligation" to "auto, 1" in config.php
Open text editor
Edit and save three times to create versions
Go to the data folder of the user in "data/$user/files_versions"
Pick one of the versions file, in my case I picked "test.txt.v1471883185"
Rename the file to "test.txt.v1371883185" (changed one digit)
update oc_filecache set name=replace('v1471883185', 'v1371883185');
(and that's the moment I started to really hate the filecache and its info redundancy)
update oc_filecache set path=replace(path, 'v1471883185', 'v1371883185');
update oc_filecache set path_hash = md5(path) where path_hash <> md5(path);
update oc_filecache set mtime=1371883185 where mtime=1471883185;
select * from oc_filecache where name like '%v1371883185'; => see the entry exists
Run occ versions:expire
select * from oc_filecache where name like '%v1371883185'; => entry is gone and file is gone from data dir

PVince81 · 2016-08-22T16:40:10Z

@DeepDiver1975 fixed a bad copy-paste mistake to make version expire work: 3d618a1

PVince81 · 2016-08-22T16:40:43Z

Both trashbin expire and version expire work now 👍

butonic · 2016-08-23T11:57:22Z

can we make https://github.com/owncloud/core/blob/master/apps/files_trashbin/lib/BackgroundJob/ExpireTrash.php#L34 reuse the new commend? same with versions? otherwise we just duplicate code.

butonic · 2016-08-23T11:59:12Z

👍 deduplicating would need further tweaking of the CommandJob. And increase complexity.
backport request to 9.0

DeepDiver1975 · 2016-08-23T12:02:54Z

deduplicating would need further tweaking of the CommandJob.

yes - there are many similarities in the way we implement occ commands, bachgroundjobs and repairsteps. We should think about implementing on single interface which can be reused in all three areas.

DeepDiver1975 · 2016-08-23T12:03:15Z

preparing backport ...

* Add occ command to expire the trashbin * Fix versions folder in setup check

DeepDiver1975 · 2016-08-23T12:04:28Z

stable9.1: [stable9.1] Add occ command to expire the trashbin (#25878) #25914
stable9: [stable9] Add occ command to expire the trashbin (#25878) #25915
stable8.2: [stable8.2] Add occ command to expire the trashbin (#25878) #25917

* Add occ command to expire the trashbin * Fix versions folder in setup check

cdamken · 2016-08-23T12:10:42Z

Nice work! 👍

* Add occ command to expire the trashbin * Fix versions folder in setup check

…d20964cfd0f7129f998b1 [stable9.1] Add occ command to expire the trashbin (#25878)

* Add occ command to expire the trashbin * Fix versions folder in setup check

…be58a2ac5542741378c25 [stable8.2] Add occ command to expire the trashbin (#25878)

lock · 2019-08-04T22:01:51Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

DeepDiver1975 added the discussion label Aug 19, 2016

DeepDiver1975 added this to the 9.2 milestone Aug 19, 2016

PVince81 reviewed Aug 22, 2016
View reviewed changes

DeepDiver1975 force-pushed the expiry-as-occ-command branch from a3d5c49 to 1c39261 Compare August 22, 2016 09:34

DeepDiver1975 added the 3 - To Review label Aug 22, 2016

DeepDiver1975 added the backport-request label Aug 22, 2016

DeepDiver1975 added 2 - Developing and removed 3 - To Review labels Aug 22, 2016

Add occ command to expire the trashbin

10fec5b

DeepDiver1975 force-pushed the expiry-as-occ-command branch from 1c39261 to 10fec5b Compare August 22, 2016 11:59

DeepDiver1975 added 3 - To Review and removed 2 - Developing labels Aug 22, 2016

PVince81 reviewed Aug 22, 2016
View reviewed changes

Fix versions folder in setup check

3d618a1

carlaschroder mentioned this pull request Aug 22, 2016

Add occ command to expire the trashbin owncloud-archive/documentation#2584

Closed

DeepDiver1975 merged commit 4f73fb4 into master Aug 23, 2016

DeepDiver1975 self-assigned this Aug 23, 2016

DeepDiver1975 deleted the expiry-as-occ-command branch August 23, 2016 12:03

DeepDiver1975 added a commit that referenced this pull request Aug 23, 2016

[stable9.1] Add occ command to expire the trashbin (#25878)

cc65604

* Add occ command to expire the trashbin * Fix versions folder in setup check

DeepDiver1975 added a commit that referenced this pull request Aug 23, 2016

[stable9] Add occ command to expire the trashbin (#25878)

1add375

* Add occ command to expire the trashbin * Fix versions folder in setup check

DeepDiver1975 added a commit that referenced this pull request Aug 23, 2016

[stable8.2] Add occ command to expire the trashbin (#25878)

5dcb684

* Add occ command to expire the trashbin * Fix versions folder in setup check

PVince81 pushed a commit that referenced this pull request Aug 24, 2016

Merge pull request #25914 from owncloud/stable9.1-4f73fb491a2bf12dbf4…

72745ef

…d20964cfd0f7129f998b1 [stable9.1] Add occ command to expire the trashbin (#25878)

PVince81 pushed a commit that referenced this pull request Aug 24, 2016

[stable9] Add occ command to expire the trashbin (#25878) (#25915)

689a7bd

* Add occ command to expire the trashbin * Fix versions folder in setup check

PVince81 pushed a commit that referenced this pull request Aug 24, 2016

Merge pull request #25917 from owncloud/stable8.2-1add375541700ef1be5…

0114b9e

…be58a2ac5542741378c25 [stable8.2] Add occ command to expire the trashbin (#25878)

butonic mentioned this pull request Aug 24, 2016

Chunked parrallel expiration jobs #25771

Closed

cdamken added the green-ticket label Aug 25, 2016

dercorn mentioned this pull request Aug 25, 2016

OC 9 - files_antivirus stable 9 owncloud/files_antivirus#98

Closed

LukasReschke mentioned this pull request Aug 29, 2016

Add occ command to expire the trashbin (#25878) nextcloud/server#1138

Merged

DeepDiver1975 removed the backport-request label Oct 8, 2016

lock bot locked as resolved and limited conversation to collaborators Aug 4, 2019

Conversation

DeepDiver1975 commented Aug 19, 2016

Uh oh!

mention-bot commented Aug 19, 2016

Uh oh!

mmattel commented Aug 19, 2016

Uh oh!

PVince81 Aug 22, 2016

Choose a reason for hiding this comment

Uh oh!

DeepDiver1975 Aug 22, 2016

Choose a reason for hiding this comment

Uh oh!

PVince81 commented Aug 22, 2016

Uh oh!

PVince81 commented Aug 22, 2016

Uh oh!

cdamken commented Aug 22, 2016

Uh oh!

cdamken commented Aug 22, 2016

Uh oh!

DeepDiver1975 commented Aug 22, 2016

Uh oh!

cdamken commented Aug 22, 2016

Uh oh!

DeepDiver1975 commented Aug 22, 2016

Uh oh!

cdamken commented Aug 22, 2016

Uh oh!

DeepDiver1975 commented Aug 22, 2016

Uh oh!

mmattel commented Aug 22, 2016

Uh oh!

PVince81 Aug 22, 2016

Choose a reason for hiding this comment

Uh oh!

PVince81 Aug 22, 2016

Choose a reason for hiding this comment

Uh oh!

DeepDiver1975 Aug 22, 2016

Choose a reason for hiding this comment

Uh oh!

VicDeo Aug 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VicDeo Aug 22, 2016

Choose a reason for hiding this comment

Uh oh!

PVince81 commented Aug 22, 2016

Uh oh!

PVince81 commented Aug 22, 2016

Uh oh!

PVince81 commented Aug 22, 2016

Uh oh!

PVince81 commented Aug 22, 2016

Uh oh!

PVince81 commented Aug 22, 2016

Uh oh!

PVince81 commented Aug 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PVince81 commented Aug 22, 2016

Uh oh!

PVince81 commented Aug 22, 2016

Uh oh!

butonic commented Aug 23, 2016

Uh oh!

butonic commented Aug 23, 2016

Uh oh!

DeepDiver1975 commented Aug 23, 2016

Uh oh!

DeepDiver1975 commented Aug 23, 2016

Uh oh!

DeepDiver1975 commented Aug 23, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cdamken commented Aug 23, 2016

Uh oh!

lock bot commented Aug 4, 2019

VicDeo Aug 22, 2016 •

edited

Loading

PVince81 commented Aug 22, 2016 •

edited

Loading

DeepDiver1975 commented Aug 23, 2016 •

edited

Loading