I work with Advanced Data Governance features in Office 365 and I was excited to see a Microsoft 365 REST API recently released to work with event-based retention. Many of the retention requirements I see with customers are event-based. (Reference: Automate event-based retention)
There are now 2 options for implementing event-based retention in Office 365:
- Option 1: Manually record the events to trigger the retention. I’ve previously blogged about this technique (Event-driven retention in Office 365. A walk-thru.), however it requires a lot of manual intervention and is really only suitable for events that happen on an infrequent basis. (Semi-annual, annual, etc.)
- Option 2 (New): Automate the event trigger by using the Microsoft 365 REST API. This can be automated with minimal/no intervention from information workers. I feel utilization of the REST API to trigger events is a big step forward in adopting event-driven retention at scale across an organization’s Office 365 content.
Practical use-cases for event-based retention using Option 2:
- Start retention on Legal case files when the litigation is complete
- Start retention on Employee files when they leave the organization
- Start retention on Project documents when the project is complete
This post will walk-thru the process for the 3rd use-case above, Project Documentation, using my own company, NexNovus Consulting, as an example.
Use-Case: Project Documents
The powerful thing about leveraging the Microsoft 365 REST API to trigger the retention event is it can be called from an assortment of source applications both inside and outside of Office 365. For purposes of this post, the event trigger will be initiated from a SharePoint list on a SharePoint Project Hub. This same SharePoint list was used to initially auto-provision each Project site (using Site Design/Site Scripts and PnP Provisioning) and so it makes sense to also use the same list to trigger retention for a Project’s documents when the project is complete.
Setup: Update Project Requests List
To do this, we need to add 2 pieces of metadata to the SharePoint list used to request a new site, Project Requests. The 2 columns are Project completion date and a unique Project Asset ID. This metadata will later be used in a Flow to trigger the retention. Below is the list settings with the 2 new site columns highlighted:
An important setting for the ProjectAssetID is to enforce unique values. In my example, I’m also limiting it to 10 characters. To my knowledge, it doesn’t matter what the Asset Id is – it just needs to uniquely identify the group of assets you want to later associate to an event. I.e. All of a project’s documents.
Follow along on these 7 high-level steps as I demonstrate how to use the above 2 columns to drive event-based retention for a Project’s documents.
Here’s a summary of the steps:
- Step 1: Create Retention event type
- Step 2: Create Retention Label(s)
- Step 3: Publish the Retention Label(s)
- Step 4: Default Retention Label settings (optional)
- Step 5: Set the Compliance Asset Id
- Step 6: Trigger the Event
- Step 7: Retention Event processing
Step 1: Create ‘Project Completion’ Event Type
Go to the Compliance Center (protection.office.com) and under Data governance menu option, select Events… Manage Event Type.This is where we’ll add the Project Completion event type.
Click +Create and enter the Name and Description for the event.
Step 2: Create a ‘NexNovus Project’ Retention Label
Still in the Compliance Center, under Classifications, select Labels and create a Retention label called NexNovus Project. We’ll set the ‘Retain or delete the content based on’ setting to ‘an event’. Select the Project Completion event type we created in Step 2. Here are the final settings for the label:
Note: you could create additional labels if different Project documents would have different retention schedules. Both could still be associated to the same retention event type, but have different retention periods. (Example: Project Charters could have a retention of 10 years past Project completion, while Project Status Reports could have a retention of 5 years past Project completion)
Step 3: Publish the Label
For this example, I’ll manually select the few existing project sites I have on this tenant as my publish locations. However, a more scalable approach would be to automatically add each Project site into the Retention Policy during the Project site auto-provisioning process using the Set-RetentionCompliancePolicy PowerShell cmdlet.
I’ve published the retention label from the previous step to my 2 Project sites (Modern Team sites). (that’s all I have right now)
Step 4: Default Retention Label Settings
Update the default retention label on each of the Project site’s document library to be NexNovus Project in Document library settings…Apply label to items in this list or library.
Once the event-based label has been applied to a document, you cannot delete the document unless you remove the label.
If you had multiple retention labels published to your Project sites, you would either pick 1 as the default at the library level, pre-create folders aligned to your retention labels in the library and default the retention label per folder OR rely on the end-user to manually assign the retention label.
Step 5: Setting the Compliance Asset Id
Every list and library has a hidden column called ComplianceAssetId. This column is what’s used to match up with events to trigger retention. As soon as the event-based retention label is assigned to a document, the ComplianceAssetId column will show as a property on the detail pane as Asset ID (circled in green):
Important: The Asset ID will be used to ensure all tagged content will be under the same event-based retention when an event (such as Project completion) happens.
Examples of Asset Id:
- For employee documents, the Asset Id might be their unique employee #
- For litigation case files, the Asset Id might be the unique legal case file #
- For project documents, the Asset Id might be a unique project #
How can we set the Compliance Asset Id for all Project documents? Unfortunately it can’t be defaulted at a site, library, or folder level. There are a couple of options:
- Option 1: Leave it up to the information worker to manually assign the correct Asset ID. I don’t love manual options due to the likelihood of error, however if you want to apply event-based retention to a small subset of project documents, this may be a sufficient option. I would not recommend this option if you want to apply an Asset ID to a larger group of project documents.
- Option 2: Automate the Asset ID on a Project’s documents. I’ve done this automation by associating a Flow to the Project document library. The Flow retrieves the unique Asset ID from the Project Requests list from Setup step above for the Project and then sets it on each Project document.
Check out my other post, Automate the Compliance Asset Id on SharePoint Documents, where I demonstrate how to update the ComplianceAssetId property thru a Microsoft Flow.
Here is a view of the Project Documents after the Flow update has run against each:Note: One of the documents above has a Retention Label associated with it that is NOT NexNovus Project (Customer Info). This is because that document has customer # information in it and is therefore under a different retention requirement. I left this document in here to demonstrate how you can have multiple retention labels associated to these documents and only some of them will be event-driven. When event processing commences for this closed project, it should detect 9 items from the above library. **See results at the end of the post!
Step 6: Project is Complete. Let’s trigger the event!
For this example, Project documents need to be retained based on the Project Completion event.
I’m triggering the event creation based on an update to a date on a SharePoint list – your trigger may be different. Regardless of how the event is triggered for your use-case, the creation of the corresponding Retention event in the Security & Compliance center can now be automated with the Microsoft 365 REST API.
Steps to create the Retention Event for Project documents:
- When the project is complete, the Project Manager will update the project list item on the Project Request list with a Project Completion Date
- A Microsoft Flow will be triggered when a Project Request list item is added/updated. It will check if the Project Completion date is entered. If it is:
- Update the (Hidden) Project Status column to ‘Closed’
- Call the Microsoft 365 REST API to create the Retention event passing in:
- Current date
- A unique Event name: I concatenated Project Completion with the Project name
- Event type: created above in Step 1. Called ‘Project Completion’
- Compliance Asset Id: the same one you used to tag the project’s documents, ProjectAssetId
- Event date: the ProjectCompletionDate converted to UTC
I won’t include all the steps in the Flow since they’re not relevant to this post. Here is the step to create the Retention event using the Microsoft 365 REST API with the parameters numbered from above:
After the Flow has executed, here is the Project Requests SharePoint list. The Flow used the ProjectAssetId and ProjectCompletionDate columns in the REST API call and updated the status to Closed:And this is the Retention Event created in the Security & Compliance Center (Data governance… Events):
Step 7: Retention Event Processing begins
Once the retention event is created, let the retaining begin!! 🙂
If you click on the event, it will show the current distribution status. It will start as Pending until a timed process runs in the background to find and deploy the retention event across all workloads where you’ve tagged content with the Asset Id (SharePoint and OneDrive), in this case ‘12345’.
Once deployed, the distribution status will change to success and it will (eventually) display counts for the content covered for the retention event. Retention will commence! For this example, it took 5 days for the item count to be updated. In this example, there were 9 documents in 1 Project site with the NexNovus Project label with Compliance Asset Id 12345:
I’ve been waiting for more automation to be introduced into the Retention and Disposition features across Office 365, something required for it to scale to the level required for many organizations I work with. Up to this point-in-time, ISVs have filled the gaps where they existed, however over time, capabilities like the one demonstrated in this post should narrow the gap.
Thanks for reading.
Photo by Karl Block on Unsplash
Thanks for this excellent post. I’ve configured event-based retention and, eventually, the event has updated to show that it has detected content against the supplied asset ID. Unfortunately, though, the retention doesn’t seem to have kicked in and disposed of the content that should now be deleted. Any idea how long the process takes to actually delete files?
Hi Ady, the process is not immediate. There is a timer process that runs in the background (weekly) to perform the actual delete. Once the deletion has been triggered for a document, you have to wait for that process to run (which you have no visibility into)
I guess you have used Basic Authentication here. When MFA is enabled, this will not work. Do you know if we can use Azure AD authentication here?
I’m sorry, I don’t know the answer to your question. I haven’t tested with all auth mechanisms.
Hi Klein, great post as always. In our specific scenario, i have a custom date column, i want the retention to kick off based on that custom column instead of using asset id, is there any way we can do this without having to add asset id ?
Hi Ganesh, you don’t have to use the built in compliance asset Id however I’ve never tried with a date column. I’ve only tried with a choice column. I’ve blogged about this here: https://joannecklein.com/2020/01/22/sharepoint-custom-metadata-and-event-based-retention/
I don’t know your use case but I’m not sure a date column would work. Try it out and see!!
Do you have a how to guide for trigging retention policy when a users is terminated?
Hello Alan, you’re not referring to an event-based retention label like what I describe in this post?
Is there any steps available for the mentioned “Start retention on Employee files when they leave the organization” i cannot seem to work this out using the steps in your guide nor can i find steps online for this you are the closest i have got to a solution.
Hi Stuart, it will follow exactly the same model. Assuming you have tagged the documents with a unique identifier for the employee as well as the retention label, you can create the event on employee departure. Please provide your scenario if you’re thinking differently.
Hi Joanne, in your example here you are able to create an event with Power Automate that doesn’t require EventTags (or labels) being passed in and I was able to get that to work (thanks!). Do you know when the New-ComplianceRetentionEvent PowerShell command will work the same way? It expects the labels to be passed in (through the -EventTags variable which cannot be blank) which is redundant when the labels are assigned to an event type when they’re created.
I do not know the answer to when the new cmdlet will work the same way. So you are wanting to trigger the event for ALL retention labels associated to the event type I assume? I get your point, however not sure if that was a miss on the product team’s part or if they intended the retention labels to always be provided regardless. Do you not know the retention labels? Or are you thinking they may change over timer? Sorry I’m not much help on this question.
Brilliant post as usual 🙂
Assume a Retention Label with some event type, which will retain documents for 5 years and then some action.
I want to run an event in which I pass the “EventDateTime” value which is in Past i.e. assume 1-Jan-2018. So, will the documents be retained until 2023 or the retention period will count from TODAY() and it will retain until 2025?
The event date you provide is used as the *start* of the retention period. This date can be the current, a past, or a future date. In your example, the retention period starts on 1-Jan-2018 and retains for 5 years from that date. You will see the event date persist in every affected file across SharePoint sites in an automatically generated column “Label Event Date”.
Is there any documentation on what permissions are required for the user account authenticating via basic auth to the REST endpoint?
I’ve created an account, set conditional access to bypass MFA, and tried adding it to any roles I can.
I cannot for the life of me get this to work. I just get the error “Resource not found for the segment ‘ComplianceRetentionEvent’.”
Hi Steven, I’m not an authentication expert. I used a Global Admin account to do this. At this link: https://docs.microsoft.com/en-us/microsoft-365/compliance/event-driven-retention?view=o365-worldwide#automate-events-by-using-a-rest-api it walks thru some steps to test this URL: https://ps.compliance.protection.outlook.com/psws/service.svc/ComplianceRetentionEvent
If you get a 302, you need to use the redirected URL instead.
Does that help?
Thanks for sharing useful article!
Do you know? is it possible to create an event not based on Event Type but based on label?
Here it seems ‘Event Type’ is required attribute, when we create event manually either we can select event type or label.
Hi, I do not know. Great question. If you find out a way do let me know please.