Updated: February 2018 [Video added]
Updated: April 2018 [SharePoint items added]
Retention. It’s not the most exciting topic in the world of Office 365, but it is a very important one. It’s the feature that ensures your organization is keeping things for as long as you should and, just as important, disposing of things as soon as you should. My focus within Office 365 is on understanding the new Retention features and how they work across Office 365 collaboration services.
Retention policies are administered from the Security and Compliance Center in Office 365 within the Data governance section. Whether the policy has been defined via a published label (Label Retention across O365) or with a label by creating one directly within this Retention section, they will all appear on this page. A Retention policy can include content from not only Exchange mailboxes, public folders, Skype conversations, SharePoint sites and OneDrive for Business content but also Office 365 Group mail and files. A retention policy is the only feature that can both retain and delete content across Office 365. As content is spread across these services, this is a “one-stop shop” for retention. A very good thing.
Check out the official Microsoft link describing what a retention policy is and when you might want to have one: Overview of retention policies.
[Update April 2018] Retention policies and labels apply to both documents and items. Added an ‘Applies to Items too!’ section.
[Update February 2018] Here’s a short video I put together on how Retention Policies and Retention Labels work together to determine the retention for a specific site/document/item. They are not mutually exclusive – you can have a retention label and a retention policy both published to the same location. This will require lots of planning by your Information Management team!
Follow along for a quick walk-thru of adding a retention policy for an Office 365 Group’s site and the ‘magic’ behind the scenes.
#1 – Setting it up
Steps to create a location-based retention policy: (“over-arching” from the video)
- Navigate to the Security and Compliance Center. (https://protection.office.com)
- Under Data Governance … Retention … click Create(+).
- Follow a 4 step wizard:
- Enter a policy name and description.
- Decide if you want to retain content for a period of time and/or delete it after a certain length of time. Content can be either retained forever or for a specified period of time based on when the content was either created or last modified. You can specify advanced retention options whereby retention is applied to content containing certain keywords or phrases or sensitive information. There will be more capabilities added for this in the future with searchable managed properties.
- Choose whether you want to preserve content from all locations (Exchange email and public folders, Skype for Business, O365 Groups, SharePoint and OneDrive for Business document libraries) or select locations. Note: you provide the Site Collection URL and not a web URL when specifying a SharePoint site to apply the policy to. (Another great reason for a Flat SharePoint architecture!)
- Optionally turn on preservation lock. This will make the policy locked preventing it from being turned off. This is what give O365 the SEC 1784 compliance certification!
A message will display stating that it can take up to 1 day for the retention policy to be applied to the locations you specified. In my experience so far it’s applied much sooner than that. However, until it’s published it will have a status of On (Pending).
#2 – Preservation Hold Library
Important thing to know about a retention policy is if there is no specific label associated with it (it is an over-arching policy), the end-user working with the content really has no idea the policy is in effect from a content editing perspective. They can continue to add/change content as they normally would, all the while the retention policy is working silently in the background ensuring the content is being preserved in another hidden location.
This hidden location? The Preservation Hold library.
You will notice although this library appears as a list in ‘Site Contents’, it behaves like a library behind-the-scenes.
When I edit the document in either of the above sites, it will allow me to edit the document, but adds an item into a special library created on the site called the Preservation Hold Library (/sites/yoursitename/PreservationHoldLibrary).
It creates the library in the site (or subsite) you are making the change in and it only creates the library when it is required (I.e. the first time a change is made and an item needs to be inserted into the library, it will create the library if it doesn’t already exist). You must be a site collection administrator to see this library.
Here is the metadata on a Preservation Hold Library (I’ve highlighted the ones that are key to the preservation policy):
Note: if a retention label has been applied to a document/item, you will NOT be able to delete it if the label is configured to retain for a certain period of time.
The end-user will receive a message (image) when they try to delete it preventing them from deleting it. Changes to a document will still appear in the Preservation Hold library.
#3 – Policy is deployed… now what?
New Content
Any content added to the site after the retention policy was put into effect will be preserved after deletion. Changes on new content aren’t copied to the Preservation Hold library the first time it’s edited, only when it’s deleted. (Unless you have versioning turned on)
For example, on my O365 group I uploaded a document (my resume in the screenshot below) after the policy was deployed. I made 2 separate edits (saved each time) and then deleted the document. Only after I deleted the document did these 3 items get added to the Preservation Hold library on the site.
Existing Content
If an item exists at the time the policy was put into effect, the first time you make a change to the document it will insert an item into the Preservation Hold Library list (what the item was before the change). Subsequent edits on the document will not insert an additional item into the list, however if the document is ever deleted, all versions of the document will be inserted into the list as separate items.
For example, if this is the version history on a document at the time of deletion:
… when the document is first edited, a snapshot of the document and its metadata as it existed prior to the change is inserted as an item in the Preservation Hold Library (the last item identified as 40 minutes ago in the diagram below). Once the document is deleted, all previous versions are inserted as items into the Preservation Hold Library:
Applies to Items too!
Retention policies (and labels) also apply to SharePoint items. If you have an over-arching retention policy applied to a list item, it follows the same rules as a document for when it’s inserted into the Preservation Hold Library. For each item entry, it stores an excel document in the library to include the metadata for the item. If the item had attachment(s), they are stored as separate documents in the Preservation Hold Library.
#4 – Removing a Preservation Hold
Content in the Preservation Hold Library will be removed when one of these things happen:
- A policy administrator has changed the rules for what’s covered by the policy and the content no longer complies.
- The policy has been disabled.
- The policy retention end date has been reached.
- If retention is based on a label, the label was removed/changed.
The retained content is not deleted immediately – this is done by a timer process.
#5 – Legacy features
Things are changing. What we used to use for Retention and Deletion is no longer recommended. Instead, Microsoft is recommending you create a Retention policy for these things: (excerpt from the Microsoft link above)
Exchange Online:
- Holds created for eDiscovery in the Security & Compliance Center (eDiscovery hold)
- In-Place Hold and Litigation Hold (eDiscovery hold)
- Retention tags and retention policies, also known as messaging records management (MRM) (Deletion only)
SharePoint Online and OneDrive for Business:
- Holds created for eDiscovery in the Security & Compliance Center (eDiscovery hold)
- Holds created in the eDiscovery Center (eDiscovery hold)
- Document deletion policies (Deletion only)
- In place records management (Retention)
- Site closure and deletion policies (Deletion only)
- Information management policies (Deletion only)
Also, if you have any legacy preservation policies defined, they will still preserve the content in the Preservation Hold Library.
SUMMARY
It is clear Microsoft is serious about Data Governance across all Office 365 services. Looking at the list of things we shouldn’t be using anymore in the Legacy features above, we need to work with Information Management teams to clear a path thru this new world of retention to help organizations get from where they are now to where they need to be.
I continue to watch this space closely as more capabilities are introduced over time.
Thanks for reading.
-JCK
Hi Joanne – good article, and timely for me as I’ve been working with Retention this week.
I wonder have you encountered the following. If a Policy includes all SharePoint sites the Exclude site option does not seem to work. This obviously causes issues if you subsequently want to remove a site – maybe to place it elsewhere due to company reorg – the site cannot be deleted, even if you placed no content there. Have a ticket open with Microsoft on this
Hi Ian. Thank you. I haven’t tried this. When I get a few minutes I will give it a try and let you know. So can I assume the policy prevents you from moving it to new location? Prevents you from renaming it?
It prevents you from deleting the site – even one just created.
Were you able to fix this?
Great piece, very practical. We’re compiling our model for content types and retention for a migration to O365. Your article makes a number of things clear. I’m just starting to look at the use of Keywords (including content types) and sensitive information types. Do you have much experience with these yet?
Hi Russ, To my knowledge, you cannot write a keyword query against the Content Type managed property from the Search Schema in SharePoint. (This is ideally what we want and something that the product team is currently working toward). Due to this, I believe the retention options are not at a point to be a 1:1 replacement for the traditional Information Management Policies we’re used to using .. It will partially depend on what your retention requirements are – if they are based on content type, I believe you are safe still doing the traditional IMP technique and I’ve been told there will be a transition piece put in place to transition from them to the new retention label model in O365. Hope this helps.
It does help, though with a bit of a sigh aimed at Microsoft… The page on labels specifies “The conditions available now support applying a label to content that contains specific words or phrases,” but I had hoped that it would extend to other kinds of Keyword queries. Both the RM model and user experienced are helped if such things work as much as possible via a single technology. So I’m glad that it is on the table.
A very helpful article. Thank you. A question if I may: I don’t want any content from our O365 site deleted automatically at any time so, when I set up my Retention Policy, I say ‘No’ to ‘Do you want us to delete it after this time.’ That much seems clear. However, what I’m not sure of is, if I say ‘No’ here, what happens to items that have been moved to the Preservation Hold Library or Recoverable Items Folder at the end of the Retention Period? Are they permanently deleted or not?
Hi David,
I apologize for this tardy response. If you don’t want the content deleted at any time, you will need to select ‘Forever’ as the retention period rather than a number of days/weeks/months/years. If you do this, you won’t get the option to ‘Do you want us to delete it after this time?’
Hope I understood your question correctly and that it helps!
Joanne K
Hi Joanne,
Thanks for your response in clarifying the nature of retention policy. I’ve read this article (http://blog.enowsoftware.com/solutions-engine/preservation-policies-in-office-365) on preservation lock, as well as other Microsoft blogs around preservation/immutability of data. It all seems to point to immutability and not permanency of data.
None of the articles that I can find explains where “legal hold” data is retained if no license is assigned to a user. I’m sure someone has to pay for the costs of storage at the least, but I can’t get clarity on 1. where this preserved user data is stored and 2. how the usage is billed if you want to keep data forever.
I recently received a response from MS partner community that “If we want to keep data forever in O365, we should keep the O365 subscription is active” – which eludes to an assignment of license to prevent data expiry – and that retained data is kept within the users allocated storage.
If you have any thoughts on this, I would appreciate it.
Best Regards,
Navs
Hi Joanne – thanks for clearing up some important misconceptions. Would you be able to guide me on how preservation/retention policies protect onedrive data after a user is deleted (and passes the 30 day data expiry)? I cannot seem to get a clear answer on this and my assumption is that the onedrive data is gone (site cleanup) after a user is deleted (after 30 days). Can you please clarify this if possible?
Hi Navs,
To my knowledge, a retention policy will not protect a User’s OneDrive for Business data past the 30 days.
There are two possible scenarios: the user account still exists, or it will be deleted…
If the user account still exists, you can use a retention policy. If the latter, OneDrive for Business content will be deleted 30 days after the account is deleted like you said. You will need to migrate the OneDrive content to a new location in order to retain it.
Another possibility is to apply a legal hold to content. Then it cannot be deleted by a user and will not be deleted even if the OneDrive account is deleted.
Hope this helps.
Joanne K
Do these retention policies work with the O365 Business Licenses, or just the Enterprise Licenses?
Hi Chris,
These retention policies also work with the O365 Business Licenses. I have an Office 365 Business Essentials tenant for my own company and they are available there.
JCK
Hi Joanne,
Above in your reply to David, you wrote “If you don’t want the content deleted at any time, you will need to select ‘Forever’ as the retention period…” I see a lot of cloud-based firms offering to backup Office 365 (email, one drive, sharepoint, etc). It seems like a short time ago that the retention policy in Office 365 was terribly short after a user deleted a file and now we can keep everything indefinitely, perhaps making 3rd party backup unnecessary unless one thinks Microsoft does not keep enough redundant copies of everything. I had been searching for backup solutions not realizing that Retention has been added as a feature. What do you think the business case(s) would be for using 3rd party backups given Retention in Office 365? Thanks!
Hi Don,
I know this is very late, but better late than never right? 🙂 I guess my opinion on 3rd party backups has changed a bit in Office 365, partly because with the proliferation of Office 365 Groups and Microsoft Teams, there is so much more a backup solution would have to consider rather than “just SharePoint” in a recover mode. That said, I guess I would ask what you are expecting out of a backup solution over and above what MSFT will provide. I wouldn’t say Retention completely covers the need as you seem to be suggesting since likely not *everything* in a tenant will be under a retention policy (although you certainly could if you wanted to).
For native recovery, there will still be reliance on the recycle bins for recovery as well as good old Microsoft support to restore sites and site collections that have been accidentally deleted. I’ve actually had to do this a few times with clients I’ve worked with and its worked great (although you do have a timeframe you have to do this in)
I don’t have any first-hand experience with 3rd party backup tools in O365 so I can’t speak to the niche they’re filling.
Hope that helps.
-JCK
Hi Joanne.
Great article…
I’ve been having trouble with Retention Policy on SharePoint Online Pages… everything works fine with document or office files, but SharePoint pages does not work.
After deleting the page, it gets into the Preservation Hold library, but they are in file format .mht and they seems empty?? What am i missing?
René
Hi Rene,
I haven’t done any testing with pages and retention so I can’t speak to the specific issue you’re experiencing. It’s interesting they’d convert them to the .mht format… but kind of makes sense as that’s a web page archive fie format. As far as it being blank, not sure. Might be worth raising a question to MSFT on that one.
Joanne K
HI Joanne,
I’m assuming the preservation hold library will count as extra storage amongst your overall SPOnline data?
Hi Lloyd, I don’t know for sure, but that’s a safe assumption.
-JCK
TK 🙂
Hi Joanne, I am very confused about O365… specifically between the Exchange retention section (the default …delete tags in retention policy) and the retention policy (retain and delete settings) under security & compliance section.
How do they interplay ? which gets to apply a label to incoming emails? Should you only have one being used? How can you tell what policies are being applied to the emails coming in?
Also how are they applied to existing emails if I change the policy? What is in-place archive versus regular archive?
Sorry for the many questions but any answers would be helpful.
thanks! Eric
Hi Eric,
I’m primarily a SharePoint person and not Exchange, however… Microsoft does recommend using the new Retention policy settings in the S&C Center instead of Retention tags in Exchange. If you had both applied, I’m not sure which one would take precedence. I would highly recommend switching over to the S&C retention model if you can. I may dig into this a bit more and write a blog post. It is very likely the S&C options are not currently as advanced as the Retention tags and policies in Exchange since the latter have been around a lot longer. Long term though, this will change and new capabilities will be introduced into the S&C only (a Unified Retention model across all Office 365 workloads – SharePoint, Exchange, Teams, etc.).
-JCK
Thank you I have come to the same conclusions after many calls with me crosoft, this is a work in progress so if possible use the newer paradigm . Thanks for the confirmation
Joanne, last august you wrote in the comments above that “I believe you are safe still doing the traditional IMP technique and I’ve been told there will be a transition piece put in place to transition from them to the new retention label model in O365.” I’ve been looking at the roadmap and uservoice but haven’t seen anything on this yet. Have you picked up any more signals on an evt. “transition piece”? Or other places where I should look?
Hi Russ, I have not heard any more on this – I will try and get some definitive direction for you. My guidance today is this… If you are doing “net new” retention configuration in SPO, I would use retention labels and retention policies. All forward investments will be in this area.
-JCK
We’re still on premise (SP2013), but looking to migrate within the coming year. We’ve got departments wanting to archive in order to be compliant (which I can arrange in SP2013), but I’m wondering whether the best advice is not to wait until we migrate (and mitigate until then) so that we can take advantage of the label and policy matrix in O365. I find it a very welcome step forward in terms of the RM aspect of the information lifecycle. But if Microsoft offers a tool for facilitating migrations from the old structures to the new, then it’s tempting to start now with SP2013, and plan toward migration.
I look forward to hearing whether there’s any movement toward a tool.
I really enjoyed the article, you have explained it so very well with screen shots and simple language. Was preparing to present a demo to one of my client. Really helped !! Thanks
Thank you! So glad it helped!
If a apply retention policy at tenant level. In case any site collection or site suppose to delete in a retention period end users will get any notifications
Hi Suresh, To my knowledge, end-users will not get notified. I have not tested this.
-JCK
Regarding “#4 – Removing a Preservation Hold”. If I “delete” a retention policy that was being applied to my sharepoint sites, will this clear out the preservation hold folder and allow me to delete the site? How long should we expect for the retention policy to be removed after deletion?
I’m dealing with modern sharepoint sites, created through O365 Groups, that complain of a compliance policy being imposed when trying to delete them. So I deleted the retention policy I “thought” was preventing this, but 4 days later and I still can’t delete the sites.
Hi Jason, I don’t know the exact time, however I *think* it’s weekly so it may not have ran yet. It’s a back-end process we have no insight into unfortunately.
-JCK
Hi Joanne. It’s good to see this topic still running. As you state, versioning is required in order to preserve a full trail of changes to a document, from its creation to deletion. If a document is evidence you need to be able to get back to the ‘original’ version even if the document has been edited subsequently. The retention policy rules are set by an ‘administrator’ in the tenancy and is something that ‘users’ cannot override, which is good. However, any group owners with site collection administration or group members with edit permission (the default) can change the document library settings, including switching off versioning. Then, without versioning on a library, deleting the document results in only the most recent ‘version’ being captured in the preservation hold library – so I’ve lost the original evidence. I suspect retention labels don’t help in that they can be removed from a document by a user. We can use retention labels with ‘classify as a record’ to ‘lock the label’ but they have to be applied by the users themselves. Am I missing something here? Any suggestions would be much appreciated?
Hi Joanne,
My question is specifically about email retention. Suppose I create a retention policy TODAY for retaining content (say up to 7 years). Will I be able to recover emails deleted BEFORE today?
Hi Adeel,
When you put a retention policy on a mailbox, it’s just like putting it on hold which means the mailbox folder assistant job stops running for that mailbox. So… as long as the emails haven’t gone thru the permanent deletion process prior to the policy being applied, you will be able to see those emails when you do a search. Does that answer your question?
-JCK
Hi Joanne,
We’ve got an issue with deleted documents showing up in Search, and the Highlighted content Web Part.
We have thousands of Site Collections that all have Site Collection owners.
If any documents are deleted, they make their way to the Preservation Hold Library via the Retention Policy – all good.
However, if one of the owners does a search for a document, all of the deleted documents show up, and if they add the Highlighted content Web Part (driven by search) and select to view mostly viewed documents from the Site, all the deleted documents show up, and can be clicked on and viewed.
How do you suggest we address this issue – can’t just be something we do for each site – needs to be a global solution.
Thanks in advance for any help you can provide.
Grant.
Hi Grant,
I’m reaching out to some contacts I have at Microsoft for some clarification around this.
-JCK
Great Article Joanne. I was also looking for policy deletion which is not captured in your article. can you explain what will happen if we delete a retention policy as well as label and then create brand new one? Will all the files that are already modified previously will still reflect old policy or will it reflect new policy now?
Hi Raj, I will have to test that scenario and document results. Likely will be a different post. Thanks for the idea!!
-JCK
Hi I have a question,
Lets say OneDrive have a file file1.doc. Retention policy is created and applied on this OneDrive user account. Therefore as per my understanding the file1.doc should be considered as Existing Content.
After retention policy is applied, If i edit File1.txt then each edited version is added in the version history.
Only after i delete the file, all versions goes to PreservationHoldLibrary.
In which case certain versions will reside in PreservationHoldLibrary and actual file is still in OneDrive user account.
Hi Diptesh,
You said you had a question, but I don’t see a question in your comment. Please elaborate.
-JCK
Great content with awesome tips. Really helpful, thanks.