Add object tagging - Moodle 310

[matthewhilton]

Summary
Allows objectfs to send tags for an object to external stores. Initially only S3 is supported. location and environment tags are added.

For technical reasons, mimetype is also stored but this is stored in metadata instead of the tags (and thus is only for new objects, migration of metadata is not feasible)

The use case for this is to be able to setup conditional lifecycle rules to send orphaned files to deep archive and eventually delete them.

See #11 (comment) for the rationale behind using S3 tags instead of other methods such as file prefixes or separate buckets.

How it works
See TAGGING.md

Cherry picks

• Cherry picked to latest branch MOODLE_402_STABLE -> Add object tagging - Moodle 42 #623

Related issues

• Closes Consider adding s3 tags for file metadata #617
• Closes Add setting to ignore backup files or manage them in a separate bucket #11

[matthewhilton]

I've been pondering the idea of removing the intermediary table from the equation here.

Originally the idea is we cache the tag values in an intermediary table, so the values can be easily queried on the moodle side for reporting. This allows you to have 'expensive' tags i.e. those that are computationally expensive to calculate, but also be able to query them very easily using the DB for reporting.

However if we want to only have 'cheap' tags, we might be able to get away with just aggregating their values in the report data table (i.e every time report is run, it recalculates the values and sums the counts together), and then instead of querying db for the cached tags, it would simply re-calculate it the moment the object is about to get updated.

The downside of this is the report generating gets a tad bit more complex and heavy because we need to constantly re-calculate the tag values (which can come from various random places in the db) as opposed to a single tag cache intermediary table.

Another downside is we wouldn't then be able to track how many have actually been tagged in the external store, whereas with an intermediary table we at least know if its in there, its value is 99% of the time the same in the external store.

I'm leaning towards the simpler solution which is just to have them in the intermediary table.

[matthewhilton]

This is mostly ready to go, will give it a test for these scenarios:

Manual testing

• New site install - enable tagging and confirm tags are uploaded when objects are replicated
• Upgraded site with existing object - enable tagging and confirm tags are uploaded via migration and also when new objects are uploaded
• Site without overwrite - confirm cannot overwrite objects and tags remain intact
• Site with overwrite - confirm can overwrite and tags are overwritten and updated
• Tagging enabled without permissions in S3 - Confirm exceptions are thrown correctly, and check API class alerts issue
• Object report - generates object report with correct tag value counts.

[bwalkerl]

Hey @matthewhilton this is looking great from the code perspective. I've pointed out a few things I've noticed, but these are mostly just some extra considerations and discussion points. I've focused more on reviewing the code and haven't tested this end to end myself - so I'm partly relying on your test results for that.

My only real concern here is the perform at scale, mostly when back filling with the adhoc task, but this is an optional feature so I think it should be fine to roll out and test as needed.

[matthewhilton]

I've not addressed all the comments yet, but just an fyi pushed an update to add a bit more visibility to the migration progress, now there is a nice table in the admin settings (for a lack of a better place to put it), let me know if you see any tweaks that should be made here, or if there are better styling options available (I couldn't find any):

Additionally pushed a CI fix, since moodle plugin ci looks to have updated on my local and was throwing errors where it wasn't before. Note the Github CI is failing due to an upstream nvm issue which i've been tracking but is not fixed yet (so it will stay red for now)

[bwalkerl]

Thanks @matthewhilton - feels like you went above and beyond with the changes so far. I've left a couple of new comments on these.

[matthewhilton]

Just marking as draft temporarily, will be adding some features we discussed internally and then will re-open it.

[matthewhilton]

Had a bit of time to work on object orphaning support.

The big issue i've found so far is that when an object is orphaned, we lose its mimetype. This is because when it goes to update the object tags after it is orphaned, the mimetype data in the mdl_files table no longer exists.

There are couple of ways to go about this:

1. Not care about mimetype anymore. Do we really need it ?
2. Push mimetype as metadata, instead of tags. This can only be done on upload (i.e. only new objects) but it is immutable so regardless of what we do with tags, the mimetype still remains. Theres no extra costs for metadata
3. Instead of the current 'replace all' tags on update, we implement some complex kind of reconciliation mechanism and only update the tags necessary. This probably has loads of edge cases.

IMO I think option 2 is the most balanced approach

[matthewhilton]

I've finished the support for tagging orphaned objects.

We now have a simple location tag which has values orphaned or active.

I split this up from the env tag since the env tag is now also used by objectfs to determine if it should overwrite tags (mainly used for lower envs that we don't want messing with prod shared objects)

I also had to move mimetype from the tags to the metadata for reasons mentioned above.