There are many things to consider when applying retention to your SharePoint Online content, and each decision you make will have follow-on effects to be aware of. I’m writing 4 posts to highlight 4 key questions to answer when configuring Office 365 retention and will identify some pros and cons of decisions surrounding each:
This post is the third in the series, Retention in SharePoint Online: The Where, What, How, and When, and answers the all-important “HOW” question:
- Part I: Retention in SharePoint Online: The “WHERE” answers the question “Where should retained content live?”
- Part II: Retention in SharePoint Online: The “WHAT” answers the question “What type of retention should be applied?”
- Part III: Retention in SharePoint Online: The “HOW” (THIS POST) answers the question “How should the retention be applied?”
- Part IV: Retention in SharePoint Online: The “WHEN” answers the question “When should the retention start?”
I think the image below sums up the retention scenario across Microsoft 365. Knowing you need it is only scratching the surface (which is partly what this blog post series is addressing)…
Let’s answer “The HOW”…
You’ve decided where you’ll retain your content and what type of retention you’ll apply, and now you need to decide how you’re going to apply it. This primarily comes down to whether or not you want to involve an end-user in applying retention, your risk tolerance, your comfort with custom code, and the license you have in your tenant. I’ll cover several options for both retention policies and retention labels to retain content stored in SharePoint Online… some automated, some manual and pros/cons for each. At the end of the post, I’ll summarize the options and the license required for each.
There are 5 ways to apply a Retention policy and 10 ways to apply a Retention label!
5 Ways to apply a Retention Policy
#1 – Org-wide all Workloads
Include all content locations in the Retention Policy configuration. You’re allowed 10 of these per tenant. There is NO LIMIT to the number of each workload that will be included in the policy. This means as new SharePoint sites and OneDrives (and mailboxes) are added to your environment, they will be automatically included and the retention policy will apply to content within.
Pros:
- simple to configure – everything falls under the same retention rules
- good model to use in tandem with other retention policies/labels across your tenant
Cons:
- there will likely be exceptions to this retention so you must ensure you are also applying more granular retention controls to handle exceptions (either thru a policy exclusion/inclusion OR a retention label)
#2 – Org-wide Workload specific
If you select Let me choose specific locations, you can choose All of one or more specific types. In this example, we could select All SharePoint sites and All Office 365 Groups to apply retention automatically to them. Similar to an org-wide retention policy, there is NO LIMIT to the number of each workload that will be included in the policy. This means as new SharePoint sites and Office 365 Groups are added to your environment, they will be automatically included and the retention policy will apply to content within
Did you know? Sites included when you toggle the SharePoint sites are Communication sites, modern team sites not backed by a Group, and classic sites. If you want to include a Modern Team site backed by a Group in retention, you must use the Office 365 groups toggle.
Pros:
- simple to configure as everything falls under the same retention rules
- good model to use in tandem with other retention policies/labels across your tenant
Cons:
- there will likely be exceptions to this retention so you must ensure you are also applying more granular retention controls to handle those exceptions (either thru a policy exclusion/inclusion OR a retention label)
#3 – Manually include/exclude specific Site/Group
Update October 2022 – consider using an adaptive scope with your retention policy to be able to include/exclude Sites/M365 Groups based on conditions. Using an adaptive scope, there are no limits.
You can manually include or exclude a SharePoint site or Office 365 Group in a Retention Policy prior to publishing. There are some limitations to this you must be aware of for the maximum number you can include or exclude per retention policy:
- 100 sites (either OneDrive or SharePoint)
- 1,000 Office 365 groups
Due to this, you must work within these bounds if you are automating a solution that includes/excludes SharePoint sites/Office 365 Groups in a retention policy.
Pros:
- simple to configure when you want to target an entire site/Group for a unique retention requirement
Cons:
- with the limits above, you may run into these in large tenants
- if doing this in an automated way, you must account for and word around the limits
#4 – Conditionally apply
You can no longer conditionally apply a retention policy to content within a location. It will always be applied to all content within the location.
Just like you can conditionally apply a retention label (you can read about this in the next section below), you can also conditionally apply a retention policy to a piece of content. At the time of this writing, the condition can be either a keyword or a sensitive information type.
This can also be done via PowerShell with the Set-RetentionComplianceRule cmdlet.
Pros:
more targeted retention so not as much storage space will be consumed by the Preservation Hold Library (PHL) on the SharePoint sitegood model to use in tandem with automation when provisioning sites and adding them into the policy
Cons:
there may be exceptions to this retention so you must ensure you are also applying more granular retention controls to handle them (either thru a policy exclusion/inclusion OR a retention label)
#5 – PowerShell
You can add a SharePoint site/Office 365 Group to a retention policy during a provisioning process. Reference: PowerShell cmdlets for Retention Policies
For a NEW Retention policy… use the New-RetentionCompliancePolicy cmdlet to create a new retention policy in the Compliance Center and associate a location to it.
For an EXISTING Retention policy… use the Set-RetentionCompliancePolicy cmdlet to update an existing Retention policy to add a new location or remove an existing location.
If you are using Azure Automation to provision a site, you could include a new site in a retention policy with the Set-RetentionCompliancePolicy cmdlet. Keep in mind the limitations identified above for the maximum number of inclusions allowed.
Pros:
- doesn’t require an Administrator to manually add a site/Group into a policy
- good model to use in tandem with site provisioning solution
Cons:
- technical debt for the custom code
10 Ways to apply a Retention Label
I’ll summarize the 10 ways, however if you’re looking for more detail, check out the link at the end of the section for a presentation where I include videos for most of the ways!
#1 – Manually apply the Retention Label
An end-user can select a document (or several documents) from a SharePoint library, open the detail pane and set the retention label. In the Apply retention label drop-down, you will only see retention labels previously published to the site. In the image below, 5 retention labels have been published to this site.
Did you know? You cannot make a retention label required. Any user with at least contribute permission level can remove a retention label or change it. An exception to this is if the retention label has made the document a record. In this case, the retention label cannot be removed or changed unless done by a Site Collection Admin (includes Group owners).
Pros:
- end-users can self-serve (a pro AND a con)
- good model to use in tandem with other retention policies/labels
Cons:
- end-users don’t always know what retention label to apply which means they may apply the wrong one or none at all
#2 – Automatically set at a library level
On every document library, there is a setting to set a default retention label. Select this and choose the retention label you would like all documents to be defaulted with when added to the library. You can (optionally) assign all existing documents this same label as well.
Did you know? Even if you don’t have the license for this feature, you will still see this as an option? Check your license before you use this feature!
Pros:
- all documents in the library inherit the label so end-users don’t have to apply it manually
- good model to use in tandem with other targeted retention labels for more targeted retention scenarios
Cons:
- often document libraries aren’t set up to align with your retention requirements
- requires an elevated level of licensing (see license table at end)
#3 – Automatically set at a folder level
Similar to applying a retention label to a document, an end-user can select a folder (or document set) from within a SharePoint library, open the detail pane and set the retention label at a folder level. In the Apply retention label drop-down, you will only see retention labels previously published to the site. In the image below, 5 retention labels have been published to this site.
Did you know? Once you apply a retention label to a folder, all documents within the folder automatically inherit the retention label (unless the document already had another retention label applied). If you remove the retention label from the folder, the retention label will also be removed from the documents within (unless a document had another retention label applied)
Pros:
- all documents in the folder inherit the label so end-users don’t have to apply it manually
- good model to use in tandem with a high-level folder structure in a library if the folders align with your retention requirements
Cons:
- requires an elevated level of licensing (see license table at end)
- end-users will be able to remove the retention label from the folder (unless it’s a record) which will also remove the retention label from documents within the folder
Auto-apply Options (#4 thru #8)
#4 thru #8 all use the auto-apply capability for a retention label. You can see this option after you’ve defined a retention label and are prompted for further actions to take. In this example, the Board record retention label has the Auto-apply a label option available.
Did you know? For any of the auto-apply options, once you define the Auto-apply condition, it can take up to 7 days for content across your tenant to have the retention label applied. Also, the content must be indexed by search! If you exclude a site or library from search, it won’t be covered in an auto-apply label policy.
#4 – Auto-apply for a sensitive information type
Select the Auto-apply a label button and then choose the first option below:
This auto-apply capability can be used to detect sensitive information types across content and apply a retention label when detected. In this example, I’m looking for Canadian Financial information to apply a retention label:
Pros:
- reduces the amount of manual intervention by end-users and therefore improves your compliance posture
Cons:
- requires an elevated level of licensing (see license table at end)
#5 – Auto-apply based on keyword query
Select the Auto-apply a label button and then choose the second option below:
A keyword query uses a special syntax called Keyword Query Language(KQL) and it can look for several things. On a SharePoint site, the most common use-case I see is Path and Site. Using these properties, you can auto-apply retention labels to a site starting with a prefix. This is something you could do if you are also controlling the provisioning process for sites, including the name. (E.g. Project sites)
Example:
This would apply a retention label to any site starting with the above URL prefix.
Reference: SharePoint searchable site properties
Pros:
- reduces the amount of manual intervention by end-users and therefore improves your compliance posture
- could use this in tandem with a site-provisioning solution since you can control a site’s URL (and other properties)
Cons:
- requires an elevated level of licensing (see license table at end)
#6 – Auto-apply based on a content type
Select the Auto-apply a label button and then choose the second option below:
This also uses KQL and allows you to target a specific content type in your environment. This is a golden opportunity to leverage any information architecture you have set up across your tenant.
Example: ContentType:”Project Document”
Did you know? You can use compound conditions in your KQL. As long as they are searchable properties, you’re gold!
Example: ContentType: AND Author:
Pros:
- reduces the amount of manual intervention by end-users and therefore improves your compliance posture
- leverages content types you may already have defined across your environment
Cons:
- requires an elevated level of licensing (see license table at end)
#7 – Auto-apply based on a metadata value
Select the Auto-apply a label button and then choose the second option below:
This also uses KQL and allows you to target a specific managed property for metadata across your environment. There are some limitations to this feature on the data types allowed. As of the time of this writing, the following column types are allowed: choice, managed metadata if published from the Content Type Hub, and Date.
This is another golden opportunity to leverage information architecture you have set up in places across your tenant for the above column types.
Example: RefinableDate##<TODAY
Did you know? For some data types, you need to map the crawled property generated from the metadata column to a predefined Refinable managed property and then use the Refinable managed property in the query.
Pros:
- reduces the amount of manual intervention by end-users and therefore improves your compliance posture
- leverages metadata you have defined across your environment
Cons:
- requires an elevated level of licensing (see license table at end)
#8 – Automatically set based on Classifier
Select the Auto-apply a label button and then choose the third option below:
You can use either one of the many built-in Classifiers or build your own to be able to intelligently detect specific types of content across data in your tenant.
“This method of classification is particularly well suited to content that isn’t easily identified by either the manual or automated pattern matching methods.”
Reference: Getting started with trainable classifiers
Defining the classifier is outside of the scope of this post, however once defined, you can use it as an auto-apply condition for a retention label.
Use-cases I can see for custom Classifiers are: employee forms, contracts, budgets, customer forms, etc.
Pros:
- reduces the amount of manual intervention by end-users and therefore improves your compliance posture
- leverages classifiers you may already have defined across your environment (classifiers can be used in other areas of Office 365)
Cons:
- requires a custom classifier to be configured in your environment (can also use a built-in one, however the most prevalent use-case I see for this is a custom classifier)
- requires an elevated level of licensing (see license table at end)
#9 – Automatically set on Modern Attachments (**PREVIEW)
Sometimes you may want to apply a retention label to the exact version of a document that was shared as a modern attachment. You can do this by selecting the fourth option below:
Once the label policy is created, any modern attachment shared in an Exchange mailbox and/or Microsoft Team you’re targeting will create a copy of the document in the hidden Preservation Hold Library on the SharePoint site where it lives within a folder called SharedVersions. It will then apply the retention label to it.
Pros:
- eDiscovery searches will be able to discern which exact version of a file was shared
#10 – Automatically set using Custom Code/PowerShell
There are several ways to set a retention label if you’re open to using some custom code or PowerShell.
Please refer to a great post by Martin Lingstuyl for a deep-dive on techniques to use thru code: Managing and applying Purview retention labels using code
SharePoint PnP (Patterns & Practices) PowerShell
You can use Set-PnPLabel to default a document library and Set-PnPListItem to set an individual document’s retention label. I have used both of these PnP cmdlets in a production environment in Azure Automation and it works well:
Set-PnPLabel -List “List name” -Label “Retention Label name” -SyncToItems $true
Set-PnPListItem –List “List name” –Identity <identity id> -Label “Retention Label name”
SharePoint CSOM (Client-Side Object Model)
You can use SPPolicyStoreProxy.SetListComplianceTag to default a retention label on a document library
**Recommend not using the ListItem.SetComplianceTag method to apply a retention label. The boolean properties required for this method are not documented and may produce inconsistent results.
For ALL custom code solutions:
Pros:
- reduces the amount of manual intervention by end-users
- can use custom logic to accommodate your unique business scenarios for applying retention
Cons:
- technical debt of custom code/solution/script and not scalable for the number of document libraries across a tenant where the code may be required
Too long, didn’t read (TLDR)…
I get it, we’re all short on time nowadays…
Here’s the list of all the ways to apply Retention Policies and Retention Labels and their licensing requirements (as of May 2020):
Please refer here for a Detailed Microsoft 365 Compliance Licensing Comparison.
Thanks for sticking with me. I hope you found this helpful, and I’d love to know if you have any other techniques you’ve come across on HOW to apply retention across SharePoint.
This Retention series covered where to store your retained content, what type of retention to apply once you put it there, how to apply it, and when to start the retention. I hope you found this series helpful and can use it to start building your own retention solutions across your tenant.
Good luck and reach out for comments, feedback, and questions!
-JCK
Hi, what does this mean to data for E1 license holders? Is retention of their different for those users?
Please refer here for a Detailed Microsoft 365 Compliance Licensing Comparison: https://docs.microsoft.com/office365/servicedescriptions/downloads/microsoft-365-compliance-licensing-comparison.pdf
-JCK
You’ve done it again. Great article.
I’d be interested in concrete examples for using trainable classifiers in this capacity (vs. other methods). I understand what they are/how they work (and how they’re used in Comms compliance) – would just like to see the business case for using them in this capacity.
Fantastic series of articles. Great session at Ignite as well. Thanks for doing this!
Thanks Chris!!
Hi Joanne, I have E3 license (no compliance add-on) and can set default labels on libraries and folders (and confirm they are being applied.).
Do you think the issue will be the retention rule will not apply even if the label is there?
Thanks and great article.
Hi Martin,
The issue will be when you “true up” your licensing with Microsoft. They don’t hide the option from you on the library. I’ve never tested if the retention rule will not apply, however I assume it will.
-JCK
Ok. So people need to be extra careful reading license details because they could end up in a bad situation if they are not aware and start using these features. I would prefer not having the options if my license does not permit to use the feature.
Thanks!
Hi Joanne, thanks so much for your brilliant articles. An IT colleague has said that multimedia files are not actioned by retention policies (and he says this is why retention cannot yet be applied to Stream). I tried to test this and to me it looks like he might be right – the label shows in the site but the videos, photos don’t show up in the compliance center. But as you have so much more experience I thought I’d ask to see if you’d come across this?
Hi Joanne, do you know what happend when you apply a retention policy based on criterias and the content criterias values change? Is it like labels where once automatically applied, it will not change even if it now meets other automatic labels criterias?
Thanks!
Hi Martin,
I don’t know from actually testing, but my gut tells me if you change the criteria, the content under retention will change. If you feel like testing that, please share back here! 🙂
-JCK
Hi Martin, I will set up some test cases to verify this and write a blog post about this.
Can you please clarify your scenario… are you saying you have a retention policy with an auto-apply condition, and there is content that matches the condition… when the content value changes so it no longer matches, you’re wondering what happens? I’ll assume this is what you mean unless I hear back from you.
-JCK
Hi Joanne,
My testing scenarios are the following (I’ll try to test these soon):
For automatically applied label policy based on KQL:
– Criteria applied to SPContentType=Invoice: All Invoice content type should be automatically labeled.
– Change the policy criteria to SPContentType=Invoice AND BusinessUnit=XYZ.
What will happend to Invoice not having BusinessUnit=XYZ? Will they keep the label?
Note: When I go back to edit an auto apply label policy, I don’t see anymore the label that was selected… why?
Also I’m wondering the same things about automatic retention policy (no labels) based on content selected on a search criteria (let say here SPContentType=Invoice. I know it does not support KQL yet but just for example clarity…) and keeps for a specific period based on ex. created date and then delete.
– If we delete Invoice before the retention period, they should be put in the PHL.
– If I change the content type of an Invoice document to another content type, then when I delete the document it will not go into the PHL?
– If we change the policy criteria to another content type, I should be able to delete Invoice and they will not go into the PHL?
Is there a way to know if the policy is applied without requiring to do some tests? I’m never sure if my tests do not work or if it is because the policies are not applied yet…
Thanks,
Martin.
This is the best article on retention policy so far .
I have question – can i create retention policy or retention label without including any sites in it and then later whenever i provision new site through site script or PnP provisioning sites ,add them into this policy.
is it possible ?
Hi YKSiddal, I’m not in front of a computer right now but i think you would have to select at least 1 location in the policy first. Just try it! You can then include the new site as a location. Watch out for the site limit on that though.
-JCK
Thank you Joanne Klein for these kinds of great articles, I always enjoy reading your articles 🙂 you are awesome.
As for retention label MS having not proper documentation If you know then can you please guide user of ListItem.SetComplianceTagWithExplicitMetasUpdate method (https://docs.microsoft.com/en-us/previous-versions/office/sharepoint-csom/mt827780(v=office.15))
We are trying to set retention label programmatically using CSOM with defined declaration date to document item. we tried using SetComplianceTagWithMetaInfo method but for this we are getting issue for some sites when we pass blockedit and block delete true. so this is again not consistent.
Nice article Joanne Klein. Really this is very helpful. I have a question regarding locations section of retention policy.
I want to apply a “3 years Retain” retention label to only 500 SharePoint sites in my tenant. So can i publish 5 retention labels with same condition and choose 100 sites in each label.?
In short, my question is
can I create 5 retention labels (each allowing for 3 year retention) and grant 100 different (500 total) sites the ability to use those labels?
Thanks
Purna
Hi Purna, yes you can do this. The retention labels need to have unique names, however the retention duration/configuration can be identical across them.
-JCK
Hi Joanne,
I would like to say that your blog posts series about Retention Label are brillant.
All your explanation and use cases get deep inside this impressive governance feature.
Could you clarify if the following scenarios are possible with E3 licenses :
– Apply a Retention Label to a folder or a Document Library in SPO
– Use a Retention Label to based on Modified date
with my E3 user (E3) , it’s possible. but I read some thread that thoses features need E5.
So I’m a bit lost.
Thanks
Fil
Hi Phil,
I’m so grateful you find the series helpful! I appreciate your feedback.
To answer your questions:
1. Although you can technically see the option in the UI to apply a default retention library at the library level (within Library settings) or set a retention label on a folder (thereby propagating the retention label to all documents within), this does require either an E5 or an advanced Compliance add-on. This is likely something that Microsoft will detect you are using.
2. It depends how you want to apply a retention label based on modified date. You’ll have to be more specific. If you set up a retention label based on modified date, you can do that with an E3 as long as you are either manually applying the retention label or setting it thru code/Power Automate Flow.
Hope that clarified it for you.
-JCK
Dear Joanne,
I have applied a retention label “ReadOnly” on a document set and it applies automatically on all documents contained in this document set, which is great.
But when I remove the retention label from the document set, it is not removed from the documents inside.
Any idea why?
Many thanks in advance and best regards,
Catherine
Hi Catherine,
When you say the retention label is ‘ReadOnly’ can I assume from this that the retention label you have configured has the ‘is a record’ checkbox checked? If so, then you have to individually remove the ‘ReadOnly’ retention label for the items within the document set. It won’t automatically remove the ‘ReadOnly’ retention label from the documents within a Document set by removing the label from the Document set. I think it was intentionally built this way as a protection mechanism so records weren’t accidentally ‘undeclared’.
Hope that helps.
-JCK
I’m curious if you have setup DLP in tandem with retention policies. My use case would be creating a reporting role to monitor admins and other users with elevated access. DLP is usually targeted towards monitoring predefined sensitive content. I’d be looking at applying it to any items with a retention policy applied to it.
My SharePoint admin experience is 2.5 years out of date and I was half way through the process of configuring an old records center setup when I realized my mistake.
Hi John, quick answer and a couple of things to consider… 1 is to ensure you publish the DLP Policy to those “elevated” accounts and the locations they use if your intention is to limit the policy to that group… You will have to include all of the workloads where your retention policy is published (Exchange, SP, OD, Teams, etc.). The other is you’re talking about a retention label policy, and NOT a container-level retention policy. You will ONLY be able to configure a DLP policy with a retention label if the workloads you’re targeting are SP and/or OD.
Hope that was helpful and will get you headed in the right direction! 🙂
-JCK
Hi Joanne. Great article. A little birdie has suggested to me that Microsoft have recently published some amended guidance that casts unwelcome ambiguity on whether we can apply retention labels via custom code/powershell within an E3 environment (#10 in your table above, dated May 2020). I know you don’t claim to be an expert in Microsoft Licencing (which sounds like a full-time job to me) but have you heard anything similar on the grapevine? Thanks
Hi Tim,
I have not heard anything; however I will make sure I update the post as soon as I do. Do you have a link to the amended guidance? 🙂 Sorry I can’t provide any more clarity at this time.
-JCK
Hi Joanne, Thanks a lot for sharing your knowledge. It has already helped us a lot in understanding retention options. Our company wants to set a default deletion policy for all documents on SharePoint for 5 years and for Teams documents for 1 year. We are thinking about the best way to do this and are testing an auto-apply retention policy based on contenttype:”Teams-Document”, but did not take effect, since two weeks have passed. We allready raised a MS support ticket for this.
Our question to you is:
1. Do you have experience with companies asking for a default deletion policy on al document on SharePoint?
2. We noticed you are using different notation in your articles in the format of configuring the condition of the auto-apply rule on contenttype. Are you able to let us know what notation should be working?
A) Is contenttype written in all lower cases?
B) How to specifying the contenttype after the colon using single quotes or double quotes?
PS:
You are using single quotes here: https://joannecklein.com/2019/07/26/auto-apply-retention-labels-in-office-365-using-content-types-and-metadata/
And double quotes int his article and here: https://medius.studios.ms/video/asset/PPT/IG19-THR2071
Thanks in advance! Eddy & Chris (the Netherlands)
Hi Eddy and Chris, this is likely worth a blog post because I can think of follow-up questions to ask you back… however I’ll address the syntax issue first. I apologize for the inconsistency I have in those examples above. If I have it in those examples above, the syntax I’m showing would’ve worked at that time… however things change and I’m reconfirming for you. I do know it doesn’t matter if you specify contenttype or ContentType before the colon, both formats have worked for me. Also, quotes of any kind are only required if you have a space in your content type name. If you have a space in the content type name, for sure double quotes work as I have a working example in my own business tenant using that format; however, I don’t have a current example in my tenant for one with single quotes – I have just now set up a test and will let you know. (I will likely blog about this as it’s an important detail that others are likely wanting to know).
To answer the other design question, that’s a big one and likely too long for here. It is an assertive stance but a good one as long as you’ve ensured you have retention labels applied to your high-value content which allows you more control. (which I’m assuming is where you’re going with the “Teams-Document” content type auto-apply?) I’d likely try to use some of the other automation if you can outside of Content Type, but that’s a factor of how complex your Retention Schedule is. I have seen organizations put in a “Delete after x years” retention policy on Exchange content (less so on SharePoint content) which I believe is mostly because they don’t have the “overall plan” for managing their records holistically across their tenant in place yet.
I’m glad you have a MS support ticket open for the issue as there may be something else going on there. (I’m assuming you don’t have a default label already applied to the documents because if you do, it won’t overwrite it) I’ll report back in this response thread with the results of my single quote testing.
-JCK
And I had the opportunity to visit your beautiful country a few times in the past few years and it was an unforgettable experience! Love the Netherlands!!! 🙂 Thanks for supporting my blog!
Hello,
I can confirm the single quotes around the content type name also works. So ContentType:’Your Custom Content Type Name’
-JCK
Hi Joanne , very helpful article but I have a question , is there a way to apply retention ( not event based retention) based on the date in the past, The scenario would be migrating documents from older SP system and I need the retention period to start from some custom date field ( not created by or modified by) for example lets say its contract expiration but the date is way in the past and I need the retention to be based on that past date, I assume the only way to do this would be through power automate. but I can’t find anything on that
I saw ur article on event based retention and using custom date with power automate , but if I have thousands of these documents I wonder how well will this work
Let me know
Thanks a lot
Hi Gary, i have so many questions to gain more clarity on the ask but I’ll do my best… I’m not aware of a way to do that using retention labels because (i don’t think) an event can be in the past. Have you tried testing that? You would have to use an event-based retention label because they only other options are starting the retention on created date, modified date, or labeled date.
Why are you concerned about “thousands of documents“? Are you saying you would have 1 event per migrated document? I think you could design your migration to possibly fill a custom column of year or quarter and then create an event to start retention for everything tagged with that year or quarter. This would be for legacy content migrated in. Would this be satisfactory for the migrated content? I should probably write a blog post on this!! 😁
Let me know how it turns out.
-JCK
Hi Joanne, thank you for a reply, Yes there basically would be an event for each document because all dates will be different. Like if i have thousands of contracts and i have a contract expiration date for each file ( the date that already occurred) using your power automate example i would have to loop through all these documents and create these event based retentions for each, which will be unmanageable
What i ended up doing is just doing custom solution using flows. but i am not using compliance center at all, i am just calculating expiration date and then have a timer job that looks for expired documents and starts a custom disposition process. This all helps me avoid getting E5 license and kind of gives me freedom to do more, there is an obvious downside to it as well with all the compliance center build in checks and balances.
I really tried to utilize the compliance center, but i don’t think it was designed to handle events that already happen
Hi Joanne – I spun up an O365 E3 trial tenant to test and demo retention policies and labels for a client. From all the documentation, it shows that I shouldn’t be able to set default Labels at the Library level or the folder level, but so far, I’m able to do both of those things with the E3 licensing.
If I check the box in Library settings to automatically apply to existing documents, it does and everything else I upload or create, gets the Library policy. If I skip that step and instead apply Labels at the folder level, everything I upload or create in the folder gets the Label. I feel like I’m missing something obvious.
I just checked the April 2021 comparison chart and it says this shouldn’t work…Any words of wisdom greatly appreciated.
Hi Kelly,
Although that advanced functionality appears and works without having the license today, that will change in the future where the capability will be blocked for usage if your tenant/users don’t have the license.
To be clear, Microsoft will be aware you are using the feature and it would likely be called out during license true-up time.
Have fun testing! It’s a great way to learn the features!
-Joanne
Thank you Joanne!!
Hi Joanne, great articles and knowledge as always, thank you.
I’ve been testing something today and came across some strange behavior and wondered if you can think of something here please or if it is something that obvious that I could be missing here…
Testing —– #9 – Automatically set using Power Automate
Scenario
Library A with default Retention Label ‘Label 1’ (retain forever) applied
This library contains document sets with a number of subfolders an files. Label 1 is automatically applied and everything is as expected, including user cannot delete anything because of the retention label.
Library B with default Retention Label ‘Label 2’ (retain 5 years) applied
This library doesn’t contain anything of its own. This will only contain documents sets (subfolders + files) when they get moved from Library A.
Library A + Power Automate Flow
Document Set(s) and subfolders/files from Library A – MOVE TO – Library B via a Power Automate Flow when the user updates one specific metadata column in Library A. This Flow is also automatically applying the Retention Label ‘Label 2’ on the Document Set(s) and sub items when they get moved to Library B.
I am doing the automatic application of Label 2 via Power Automate as, just moving the items from Library A to Library B will not replace the retention label from ‘Label1’ to ‘Label 2’ for those items. (OOTB behavior I believe)
Everything is Moving + Retention Label is getting applied via Flow as expected.
Strange Behavior
User when tries to delete the items in Library B (Doc Set, Subfolder or Docs), the ‘Label 2’ retention is not blocking that operation and the items are getting deleted and are moved to the Bin.
To make sure the ‘Label 2’ is working as it should, I have tested by creating new items from scratch within Library B and default ‘Label 2’ for that library gets applied automatically as expected. ‘Label 2’ on this item is blocking the Delete operation as expected.
Any words of wisdom from you on this will be greatly appreciated! Thanks
Hi Sumit,
I’ve never experienced this. Where my “troubleshooting” brain goes is the attributes you’re setting in the PA Flow when you’re applying the ‘Label 2’. There are 4 of them – one of which is “isTagPolicyHold” – like I said in my blog post I don’t know exactly what this does, but you can try playing with those settings.
It sounds like you’ve done your due diligence on testing. If you adjust those settings and still can’t get it to work, then I’d open up an official ticket with MSFT.
If you get it to work, please reply back with your solution so everyone can benefit from your learning. Thanks!
-JCK
Thanks, Joanne
Appreciate your response.
Other thing I checked was the confirmation via PowerShell that the label is definitely applied to the items.
Also been testing removal of labels via PA flow to Doc sets with sub folders, and, Folders with subfolders etc. In that scenario, it doesn’t seem to like pushing inheritance through to the files/docs from the top level (doc set OR Folder), so all individual files inside the containers still keep the label applied to them, including visually.
Anyway, still in it so I’ll be testing those attributes that you mention. 2 of those are obvious ones, as you’ve noted in your article too. The IsSuperLock sounds like it might be the Regulatory Record but will check to make sure.
It’s a shame that Microsoft don’t seem to have any documentation on these whatsoever.
I’m assuming the automation part via flow in Event based retention will also be impacted by this strange behaviour that I’ve come across – labels appear visually but don’t work as expected.
Thanks for your response again and will share if any interesting findings.
Sumit
Also worth mentioning here… I’m making an assumption the label you’re trying to remove at the Doc set level is NOT a record label. Although you can apply a retention label that marks an item as a record at a Doc Set or folder level and it nicely propagates down to the files within, the same is NOT true when you remove the label at the Doc Set/folder level.
Thought I’d mention that in case that’s your scenario.
-JCK
That’s correct, a standard retention label in my scenarios.
Thanks for pointing it out though!
Failing the expected application of retention label via Flow, the idea is to reverse it around, and remove labels before moving items to another library, where they will then get the default label applied at the new library (Doesn’t work if the original labels still applied to the items). Wanted to get this to also work by just removing labels on the top level folder or even including subfolders, but seems no propagation, so removal of retention labels may also require looping through each individual item.
First some fun with TMR video files auto labelling/retention after the shift from Stream to ODSP, and now this, I’m almost starting to enjoy how MS are helping us out here 🙂
Sumit
Hi Joanne,
Im trying to setup an auto retention policy to Auto-apply based on keyword query. I have a choice column and have it all mapped to my refinablestring00: sting and the string is searchable via the site, however the label never gets auto-applied applied after 7 days. I’ve also tried creating the policy again as per the MS documentation but it never works. Do you have any suggestions?
Also when applying and Auto-apply policy, are you able to drill it down to document library’s or can it only be done at site level? I have a few document library’s in one site but only want the rule to apply to 2/4 sites. I was going to simply to chanced the crawled properly but them seem to have used the same name.
Hi Curtis,
Hard to troubleshoot over this. Sounds like you’re doing it correctly. Is the example you’ve provided me accurate? Meaning… are you using a RefinableString MP between 00 and 99? Also, did you wait 1 day after doing the mapping before setting up the policy (that is a new requirement I recently read) ALso, what is the retention label based off of… is it an event-based label by any chance?
Your second question… if all of those libraries are on the same site, you might have to use the DocumentLink property to reference the 2 libraries you want to include in your KQL. Without that exclusion, it will look for a match in the Search index across the location (site) you have it published to. Link: https://docs.microsoft.com/en-us/microsoft-365/compliance/keyword-queries-and-search-conditions?view=o365-worldwide#searchable-site-properties
Hope this helps.
-JCK
Hi Joanne,
Yes the example provided is accurate. I have used refinablestring00 and mapped it to the crawled property of the choice column called Document Type on the library. If I where to search refinablestring00:CurtisTest it will display all documents with the document type as CurtisTest. I cant remember If I waited a day after mapping. Is it worth remapping to refinablestring01 and waiting a day then setting up the policy? The label is based off a choice column called document type.
Amazing thanks will give that a try. Im guessing if I where to use multiple property’s to reference I just need ot put them on separate lines?
Hi Curtis, a couple of things. Yes, it’s worth remapping… it’s what I would do. I’ve also learned that just because your query works on the Microsoft search box from the site doesn’t 100% guarantee the query will work in the auto-apply condition. You are limited to RefinableString00-99 in the auto-apply but you can go above RefinableString100 in the Microsoft search box. Also, please propercase your RefinableString mp.
Multiple properties can use OR and AND in a KQL query. They don’t have to be on separate lines.
Please let me know if this helps.
-Joanne
Hello Joanne
I had a query on Power Automate applying Retention Labels.
I’ve run tests in my demo environment and found that when I follow the steps outlined then the labels are applied successfully but I’m still able to delete data even though it’s tagged with a label. My environment just allows a full delete with no PHL – maybe the upcoming change for deleting data with a label would create the PHL at this point?
Through PowerShell (CAML query/PnP) or using the Label Setting column I can see the value of 2 when running Power Automate, I’m assuming this maps to the Compliance Flag – finding documentation on this is difficult to say the least!
I then manually applied a label to content through the SharePoint UI and found this had a Compliance Flag/Label Setting of 3, I’m unable to delete data using this approach.
I then changed the value for isTagPolicyHold to ‘true’ and ran Power Automate again, this then had a Compliance Flag/Label Setting of 3 and I’m unable to delete data as the label prevents it, consistent with manually applying the label.
I think isTagPolicyHold maps to BlockDelete which is consistent with how manually applying the label behaves, or at least my current understanding of things given how limited the documentation is.
I’m monitoring Compliance Centre to look for how these labels are reported on through Reports/Data Classification, so far they’re showing as manually applied labels. If isTagPolicyHold did map to a disposition review would this show through Records Management or maybe somewhere else?
Interestingly, I also changed isTagPolicyRecord to ‘true’ and that creates a Locked Record with a Compliance Flag/Label Setting of 646.
Outside of this, I’ve learnt a lot from following your posts so it is very much appreciated.
Thanks.
Matt.
Hello Joanne,
Love this series of very helpful posts on building retention into SharePoint!
I think I have the same issue as Gary above. I am migrating a series of document libraries from a non-SP container into SP. These libraries all have doc sets based on year complete, since our retention rules start at that point. So, if I understand from Gary’s point above, there is no way to link a backdate to a retention label. So I would not be able to link a retention label directly to a metadata column of “Year” found throughout all libraries/doc sets on the site, and have it add a 60 year value automatically? Could this be done through another method? [Very much a noobie here]
Since the document sets themselves are complete as loaded, would I then need to develop a series of labels with differing years that would then add up to the 60 to disposition rule? For instance if the document set is Year 1982 it already has forty years of retention, so I would build a record label for that doc set for 20 year retention going forward?
Then going forward for this year’s document set onward, it would just simply be the 60 year label? Would this be a good method to handle backdating, or can the metadata label drive the retention label at all?
Thanks so much,
Sue
Hi Sue,
To address your question, I’m testing something to backdate the start of the retention period. Due to timing, I won’t see the results for up to 14 days. To be clear, it will require an event-based retention label so you can control the date.
The second alternative you suggest would work; however, I’m testing out a method that won’t require you to do that.
I will be writing another post for this.
-Joanne
Hi Sue, please check out my recent post where I walk through the steps to accomplish this via one of two options. Hope it helps!
Link: https://joannecklein.com/2022/04/08/back-dating-retention-in-microsoft-365/
-Joanne
Hi Joanne,
You mention “There is NO LIMIT to the number of each workload that will be included in the policy. This means as new SharePoint sites and OneDrives (and mailboxes) are added to your environment, they will be automatically included and the retention policy will apply to content within.”
However, there are some workload limitations mentioned here, https://learn.microsoft.com/en-us/purview/retention-limits
“Within this 10,000 policies limit, there are also some limits on the maximum number of policies for retention per workload:
Exchange (any configuration): 1,800
Per mailbox: 25 is the recommended maximum before performance might be impacted; 50 is the supported limit.
SharePoint or OneDrive: (all sites automatically included): 13
SharePoint or OneDrive (specific locations included or excluded): 2,600”
What exactly does this mean?
Does it perhaps mean that we can only have 13 policies that include all sites? And all other policies must specify sites and match less than 100?
Also, in this quoted section there isn’t a reference to Microsoft 365 groups (teams sites) so not sure if this “13” value applies to them.
It’s a bit confusing the wording “workload” limitation plus the per-policy limitation.
Can you please clarify what are the real, up-to-date limitations, particularly for SharePoint sites based on Microsoft 365 groups? (which btw is another wording complication since they are all sites but somehow are bound to different rules)
I suppose there could also have been changes since your article was first written.
Thanks in advance.
Hi Joanne,
You mention “There is NO LIMIT to the number of each workload that will be included in the policy. This means as new SharePoint sites and OneDrives (and mailboxes) are added to your environment, they will be automatically included and the retention policy will apply to content within.”
However, there are some workload limitations mentioned here, https://learn.microsoft.com/en-us/purview/retention-limits
“Within this 10,000 policies limit, there are also some limits on the maximum number of policies for retention per workload:
Exchange (any configuration): 1,800
Per mailbox: 25 is the recommended maximum before performance might be impacted; 50 is the supported limit.
SharePoint or OneDrive: (all sites automatically included): 13
SharePoint or OneDrive (specific locations included or excluded): 2,600”
What exactly does this mean?
Does it perhaps mean that we can only have 13 policies that include all sites? And all other policies must specify sites and match less than 100?
Also, in this quoted section there isn’t a reference to Microsoft 365 groups (teams sites) so not sure if this “13” value applies to them.
It’s a bit confusing the wording “workload” limitation plus the per-policy limitation.
Can you please clarify what are the real, up-to-date limitations, particularly for SharePoint sites based on Microsoft 365 groups? (which btw is another wording complication since they are all sites but somehow are bound to different rules)
I suppose there could also have been changes since your article was first written.
Thanks in advance.
(not sure if previous comment went trough, please delete if duplicate)