Automate Event-based Retention in Office 365

Reading Time: 8 minutes

I work with Advanced Data Governance features in Office 365 and I was excited to see a Microsoft 365 REST API recently released to work with event-based retention. Many of the retention requirements I see with customers are event-based. (Reference: Automate event-based retention)

There are now 2 options for implementing event-based retention in Office 365:

• Option 1: Manually record the events to trigger the retention. I’ve previously blogged about this technique (Event-driven retention in Office 365. A walk-thru.), however it requires a lot of manual intervention and is really only suitable for events that happen on an infrequent basis. (Semi-annual, annual, etc.)
• Option 2 (New): Automate the event trigger by using the Microsoft 365 REST API. This can be automated with minimal/no intervention from information workers. I feel utilization of the REST API to trigger events is a big step forward in adopting event-driven retention at scale across an organization’s Office 365 content.

Practical use-cases for event-based retention using Option 2:

• • Start retention on Legal case files when the litigation is complete
  • Start retention on Employee files when they leave the organization
  • Start retention on Project documents when the project is complete

This post will walk-thru the process for the 3rd use-case above, Project Documentation, using my own company, NexNovus Consulting, as an example.

Use-Case: Project Documents

The powerful thing about leveraging the Microsoft 365 REST API to trigger the retention event is it can be called from an assortment of source applications both inside and outside of Office 365. For purposes of this post, the event trigger will be initiated from a SharePoint list on a SharePoint Project Hub. This same SharePoint list was used to initially auto-provision each Project site (using Site Design/Site Scripts and PnP Provisioning) and so it makes sense to also use the same list to trigger retention for a Project’s documents when the project is complete.

Setup: Update Project Requests List

To do this, we need to add 2 pieces of metadata to the SharePoint list used to request a new site, Project Requests. The 2 columns are Project completion date and a unique Project Asset ID. This metadata will later be used in a Flow to trigger the retention. Below is the list settings with the 2 new site columns highlighted:

An important setting for the ProjectAssetID is to enforce unique values. In my example, I’m also limiting it to 10 characters. To my knowledge, it doesn’t matter what the Asset Id is – it just needs to uniquely identify the group of assets you want to later associate to an event. I.e. All of a project’s documents.

---

Follow along on these 7 high-level steps as I demonstrate how to use the above 2 columns to drive event-based retention for a Project’s documents.

Here’s a summary of the steps:

• • Step 1: Create Retention event type
  • Step 2: Create Retention Label(s)
  • Step 3: Publish the Retention Label(s)
  • Step 4: Default Retention Label settings (optional)
  • Step 5: Set the Compliance Asset Id
  • Step 6: Trigger the Event
  • Step 7: Retention Event processing

---

Step 1: Create ‘Project Completion’ Event Type

Go to the Compliance Center (protection.office.com) and under Data governance menu option, select Events… Manage Event Type.This is where we’ll add the Project Completion event type.

Click +Create and enter the Name and Description for the event.

---

Step 2: Create a ‘NexNovus Project’ Retention Label

Still in the Compliance Center, under Classifications, select Labels and create a Retention label called NexNovus Project. We’ll set the ‘Retain or delete the content based on’ setting to ‘an event’. Select the Project Completion event type we created in Step 2. Here are the final settings for the label:

Note: you could create additional labels if different Project documents would have different retention schedules. Both could still be associated to the same retention event type, but have different retention periods. (Example: Project Charters could have a retention of 10 years past Project completion, while Project Status Reports could have a retention of 5 years past Project completion)

---

Step 3: Publish the Label

For this example, I’ll manually select the few existing project sites I have on this tenant as my publish locations. However, a more scalable approach would be to automatically add each Project site into the Retention Policy during the Project site auto-provisioning process using the Set-RetentionCompliancePolicy PowerShell cmdlet.

I’ve published the retention label from the previous step to my 2 Project sites (Modern Team sites). (that’s all I have right now)

---

Step 4: Default Retention Label Settings

Update the default retention label on each of the Project site’s document library to be NexNovus Project in Document library settings…Apply label to items in this list or library.

Once the event-based label has been applied to a document, you cannot delete the document unless you remove the label.

If you had multiple retention labels published to your Project sites, you would either pick 1 as the default at the library level, pre-create folders aligned to your retention labels in the library and default the retention label per folder OR rely on the end-user to manually assign the retention label.

---

Step 5: Setting the Compliance Asset Id

Every list and library has a hidden column called ComplianceAssetId. This column is what’s used to match up with events to trigger retention. As soon as the event-based retention label is assigned to a document, the ComplianceAssetId column will show as a property on the detail pane as Asset ID (circled in green):

Important: The Asset ID will be used to ensure all tagged content will be under the same event-based retention when an event (such as Project completion) happens.

Examples of Asset Id:

• For employee documents, the Asset Id might be their unique employee #
• For litigation case files, the Asset Id might be the unique legal case file #
• For project documents, the Asset Id might be a unique project #

How can we set the Compliance Asset Id for all Project documents? Unfortunately it can’t be defaulted at a site, library, or folder level. There are a couple of options:

• Option 1: Leave it up to the information worker to manually assign the correct Asset ID. I don’t love manual options due to the likelihood of error, however if you want to apply event-based retention to a small subset of project documents, this may be a sufficient option. I would not recommend this option if you want to apply an Asset ID to a larger group of project documents.
• Option 2: Automate the Asset ID on a Project’s documents.  I’ve done this automation by associating a Flow to the Project document library. The Flow retrieves the unique Asset ID from the Project Requests list from Setup step above for the Project and then sets it on each Project document.

Check out my other post, Automate the Compliance Asset Id on SharePoint Documents, where I demonstrate how to update the ComplianceAssetId property thru a Microsoft Flow.

Here is a view of the Project Documents after the Flow update has run against each:Note: One of the documents above has a Retention Label associated with it that is NOT NexNovus Project (Customer Info). This is because that document has customer # information in it and is therefore under a different retention requirement. I left this document in here to demonstrate how you can have multiple retention labels associated to these documents and only some of them will be event-driven. When event processing commences for this closed project, it should detect 9 items from the above library. **See results at the end of the post!

---

Step 6: Project is Complete. Let’s trigger the event!

For this example, Project documents need to be retained based on the Project Completion event.

I’m triggering the event creation based on an update to a date on a SharePoint list – your trigger may be different. Regardless of how the event is triggered for your use-case, the creation of the corresponding Retention event in the Security & Compliance center can now be automated with the Microsoft 365 REST API.

Steps to create the Retention Event for Project documents:

1. When the project is complete, the Project Manager will update the project list item on the Project Request list with a Project Completion Date
2. A Microsoft Flow will be triggered when a Project Request list item is added/updated. It will check if the Project Completion date is entered. If it is:
   • Update the (Hidden) Project Status column to ‘Closed’
   • Call the Microsoft 365 REST API to create the Retention event passing in:
     1. Current date
     2. A unique Event name: I concatenated Project Completion with the Project name
     3. Event type: created above in Step 1. Called ‘Project Completion’
     4. Compliance Asset Id: the same one you used to tag the project’s documents, ProjectAssetId
     5. Event date: the ProjectCompletionDate converted to UTC

I won’t include all the steps in the Flow since they’re not relevant to this post. Here is the step to create the Retention event using the Microsoft 365 REST API with the parameters numbered from above:

After the Flow has executed, here is the Project Requests SharePoint list. The Flow used the ProjectAssetId and ProjectCompletionDate columns in the REST API call and updated the status to Closed:And this is the Retention Event created in the Security & Compliance Center (Data governance… Events):

---

Step 7: Retention Event Processing begins

Once the retention event is created, let the retaining begin!! 🙂

If you click on the event, it will show the current distribution status. It will start as Pending until a timed process runs in the background to find and deploy the retention event across all workloads where you’ve tagged content with the Asset Id (SharePoint and OneDrive), in this case ‘12345’.

Once deployed, the distribution status will change to success and it will (eventually) display counts for the content covered for the retention event. Retention will commence! For this example, it took 5 days for the item count to be updated. In this example, there were 9 documents in 1 Project site with the NexNovus Project label with Compliance Asset Id 12345:

---

My thoughts

I’ve been waiting for more automation to be introduced into the Retention and Disposition features across Office 365, something required for it to scale to the level required for many organizations I work with. Up to this point-in-time, ISVs have filled the gaps where they existed, however over time, capabilities like the one demonstrated in this post should narrow the gap.

Thanks for reading.

-JCK

---

Photo by Karl Block on Unsplash