Retention in Office 365. The new way.

Blog post: 3 minute read.
Last updated: February 2018 [Video added]

Retention. It’s not the most exciting topic in the world of Office 365, but it is a very important one. It’s the feature that ensures your organization is keeping things for as long as you should and, just as important, disposing of things as soon as you should. My focus within Office 365 is on understanding the new Retention features and how they work across Office 365 collaboration services.

Data Retention MenuRetention policies are administered from the Security and Compliance Center in Office 365 within the Data governance section. Whether the policy has been defined via a published label (Label Retention across O365) or by creating one directly within this Retention section, they will all appear on this page. A Retention policy can include content from not only Exchange mailboxes, public folders, Skype conversations, SharePoint sites and OneDrive for Business content but also Office 365 Group mail and files. A retention policy is the only feature that can both retain and delete content across Office 365. As content is spread across these services, this is a “one-stop shop” for retention. A very good thing.

Check out the official Microsoft link describing what a retention policy is and when you might want to have one: Overview of retention policies.

[Update February 2018] Here’s a short video I put together on how Retention Policies and Retention Labels work together to determine the retention for a specific site/document. This will require lots of planning by your Information Management team!

Follow along for a quick walk-thru of adding a retention policy for an Office 365 Group’s site and the ‘magic’ behind the scenes.


#1 – Setting it up

Steps to create a location-based retention policy: (“over-arching” from the video)

  1. Navigate to the Security and Compliance Center. (https://protection.office.com)
  2. Under Data Governance … Retention … click Create(+).
  3. Follow a 4 step wizard:
    1. Enter a policy name and description.
    2. Decide if you want to retain content for a period of time and/or delete it after a certain length of time. Content can be either retained forever or for a specified period of time based on when the content was either created or last modified. You can specify advanced retention options whereby retention is applied to content containing certain keywords or phrases or sensitive information. There will be more capabilities added for this in the future with searchable managed properties.
    3. Choose whether you want to preserve content from all locations (Exchange email and public folders, Skype for Business, O365 Groups, SharePoint and OneDrive for Business document libraries) or select locations.   Note: you provide the Site Collection URL and not a web URL when specifying a SharePoint site to apply the policy to. (Another great reason for a Flat SharePoint architecture!)
    4. Optionally turn on preservation lock. This will make the policy locked preventing it from being turned off. This is what give O365 the SEC 1784 compliance certification!

A message will display stating that it can take up to 1 day for the retention policy to be applied to the locations you specified. In my experience so far it’s applied much sooner than that. However, until it’s published it will have a status of On (Pending).


#2 – Preservation Hold Library

Important thing to know about a retention policy is regardless of how it was applied (by either classifying data with labels or by applying a policy to a location as was done in step 1 above) the end-user working with the content really has no idea the policy is in effect from a content editing perspective. They can continue to add/change content as they normally would, all the while the retention policy is working silently in the background ensuring the content is being preserved in another hidden location.

This hidden location? The Preservation Hold library.

When I edit the document in either of the above sites, it will allow me to edit the document, but adds an item into a special library created on the site called the Preservation Hold Library (/sites/yoursitename/PreservationHoldLibrary).

It creates the library in the site (or subsite) you are making the change in and it only creates the library when it is required (I.e. the first time a change is made and an item needs to be inserted into the library, it will create the library if it doesn’t already exist). You must be a site collection administrator to see this library.

Here is the metadata on a Preservation Hold Library (I’ve highlighted the ones that are key to the preservation policy):

preservationholdlibrarycolumns


#3 – Policy is deployed… now what?

New Content

Any content added to the site after the retention policy was put into effect will be preserved after deletion. Changes on new content aren’t copied to the Preservation Hold library the first time it’s edited, only when it’s deleted. (Unless you have versioning turned on)

For example, on my O365 group I uploaded a document (my resume in the screenshot below) after the policy was deployed. I made 2 separate edits (saved each time) and then deleted the document. Only after I deleted the document did these 3 items get added to the Preservation Hold library on the site.

NetNewDocumentDeletyed


Existing Content

If an item exists at the time the policy was put into effect, the first time you make a change to the document it will insert an item into the Preservation Hold Library list (what the item was before the change). Subsequent edits on the document will not insert an additional item into the list, however if the document is ever deleted, all versions of the document will be inserted into the list as separate items.

For example, if this is the version history on a document at the time of deletion:

FileVersionHistory

… when the document is first edited, a snapshot of the document and its metadata as it existed prior to the change is inserted as an item in the Preservation Hold Library (the last item identified as 40 minutes ago in the diagram below). Once the document is deleted, all previous versions are inserted as items into the Preservation Hold Library:

PreservationHoldLibraryAfterDelete


#4 – Removing a Preservation Hold

Content in the Preservation Hold Library will be removed when one of these things happen:

  1. A policy administrator has changed the rules for what’s covered by the policy and the content no longer complies.
  2. The policy has been disabled.
  3. The policy retention end date has been reached.
  4. If retention is based on a label, the label was removed/changed.

The retained content is not deleted immediately – this is done by a timer process.


#5 – Legacy features

Things are changing. What we used to use for Retention and Deletion is no longer recommended. Instead, Microsoft is recommending you create a Retention policy for these things: (excerpt from the Microsoft link above)

Exchange Online:

SharePoint Online and OneDrive for Business:

Also, if you have any legacy preservation policies defined, they will still preserve the content in the Preservation Hold Library.


SUMMARY

It is clear Microsoft is serious about Data Governance across all Office 365 services. Looking at the list of things we shouldn’t be using anymore in the Legacy features above, we need to work with Information Management teams to clear a path thru this new world of retention to help organizations get from where they are now to where they need to be.

I continue to watch this space closely as more capabilities are introduced over time.

Thanks for reading.

-JCK

21 comments

  1. Hi Joanne – good article, and timely for me as I’ve been working with Retention this week.

    I wonder have you encountered the following. If a Policy includes all SharePoint sites the Exclude site option does not seem to work. This obviously causes issues if you subsequently want to remove a site – maybe to place it elsewhere due to company reorg – the site cannot be deleted, even if you placed no content there. Have a ticket open with Microsoft on this

    1. Hi Ian. Thank you. I haven’t tried this. When I get a few minutes I will give it a try and let you know. So can I assume the policy prevents you from moving it to new location? Prevents you from renaming it?

  2. Great piece, very practical. We’re compiling our model for content types and retention for a migration to O365. Your article makes a number of things clear. I’m just starting to look at the use of Keywords (including content types) and sensitive information types. Do you have much experience with these yet?

    1. Hi Russ, To my knowledge, you cannot write a keyword query against the Content Type managed property from the Search Schema in SharePoint. (This is ideally what we want and something that the product team is currently working toward). Due to this, I believe the retention options are not at a point to be a 1:1 replacement for the traditional Information Management Policies we’re used to using .. It will partially depend on what your retention requirements are – if they are based on content type, I believe you are safe still doing the traditional IMP technique and I’ve been told there will be a transition piece put in place to transition from them to the new retention label model in O365. Hope this helps.

  3. It does help, though with a bit of a sigh aimed at Microsoft… The page on labels specifies “The conditions available now support applying a label to content that contains specific words or phrases,” but I had hoped that it would extend to other kinds of Keyword queries. Both the RM model and user experienced are helped if such things work as much as possible via a single technology. So I’m glad that it is on the table.

  4. A very helpful article. Thank you. A question if I may: I don’t want any content from our O365 site deleted automatically at any time so, when I set up my Retention Policy, I say ‘No’ to ‘Do you want us to delete it after this time.’ That much seems clear. However, what I’m not sure of is, if I say ‘No’ here, what happens to items that have been moved to the Preservation Hold Library or Recoverable Items Folder at the end of the Retention Period? Are they permanently deleted or not?

    1. Hi David,
      I apologize for this tardy response. If you don’t want the content deleted at any time, you will need to select ‘Forever’ as the retention period rather than a number of days/weeks/months/years. If you do this, you won’t get the option to ‘Do you want us to delete it after this time?’

      Hope I understood your question correctly and that it helps!
      Joanne K

      1. Hi Joanne,

        Thanks for your response in clarifying the nature of retention policy. I’ve read this article (http://blog.enowsoftware.com/solutions-engine/preservation-policies-in-office-365) on preservation lock, as well as other Microsoft blogs around preservation/immutability of data. It all seems to point to immutability and not permanency of data.

        None of the articles that I can find explains where “legal hold” data is retained if no license is assigned to a user. I’m sure someone has to pay for the costs of storage at the least, but I can’t get clarity on 1. where this preserved user data is stored and 2. how the usage is billed if you want to keep data forever.

        I recently received a response from MS partner community that “If we want to keep data forever in O365, we should keep the O365 subscription is active” – which eludes to an assignment of license to prevent data expiry – and that retained data is kept within the users allocated storage.

        If you have any thoughts on this, I would appreciate it.

        Best Regards,
        Navs

  5. Hi Joanne – thanks for clearing up some important misconceptions. Would you be able to guide me on how preservation/retention policies protect onedrive data after a user is deleted (and passes the 30 day data expiry)? I cannot seem to get a clear answer on this and my assumption is that the onedrive data is gone (site cleanup) after a user is deleted (after 30 days). Can you please clarify this if possible?

    1. Hi Navs,
      To my knowledge, a retention policy will not protect a User’s OneDrive for Business data past the 30 days.
      There are two possible scenarios: the user account still exists, or it will be deleted…

      If the user account still exists, you can use a retention policy. If the latter, OneDrive for Business content will be deleted 30 days after the account is deleted like you said. You will need to migrate the OneDrive content to a new location in order to retain it.

      Another possibility is to apply a legal hold to content. Then it cannot be deleted by a user and will not be deleted even if the OneDrive account is deleted.

      Hope this helps.
      Joanne K

    1. Hi Chris,
      These retention policies also work with the O365 Business Licenses. I have an Office 365 Business Essentials tenant for my own company and they are available there.

      JCK

  6. Hi Joanne,
    Above in your reply to David, you wrote “If you don’t want the content deleted at any time, you will need to select ‘Forever’ as the retention period…” I see a lot of cloud-based firms offering to backup Office 365 (email, one drive, sharepoint, etc). It seems like a short time ago that the retention policy in Office 365 was terribly short after a user deleted a file and now we can keep everything indefinitely, perhaps making 3rd party backup unnecessary unless one thinks Microsoft does not keep enough redundant copies of everything. I had been searching for backup solutions not realizing that Retention has been added as a feature. What do you think the business case(s) would be for using 3rd party backups given Retention in Office 365? Thanks!

    1. Hi Don,
      I know this is very late, but better late than never right? 🙂 I guess my opinion on 3rd party backups has changed a bit in Office 365, partly because with the proliferation of Office 365 Groups and Microsoft Teams, there is so much more a backup solution would have to consider rather than “just SharePoint” in a recover mode. That said, I guess I would ask what you are expecting out of a backup solution over and above what MSFT will provide. I wouldn’t say Retention completely covers the need as you seem to be suggesting since likely not *everything* in a tenant will be under a retention policy (although you certainly could if you wanted to).

      For native recovery, there will still be reliance on the recycle bins for recovery as well as good old Microsoft support to restore sites and site collections that have been accidentally deleted. I’ve actually had to do this a few times with clients I’ve worked with and its worked great (although you do have a timeframe you have to do this in)

      I don’t have any first-hand experience with 3rd party backup tools in O365 so I can’t speak to the niche they’re filling.

      Hope that helps.
      -JCK

  7. Hi Joanne.
    Great article…
    I’ve been having trouble with Retention Policy on SharePoint Online Pages… everything works fine with document or office files, but SharePoint pages does not work.
    After deleting the page, it gets into the Preservation Hold library, but they are in file format .mht and they seems empty?? What am i missing?

    René

    1. Hi Rene,
      I haven’t done any testing with pages and retention so I can’t speak to the specific issue you’re experiencing. It’s interesting they’d convert them to the .mht format… but kind of makes sense as that’s a web page archive fie format. As far as it being blank, not sure. Might be worth raising a question to MSFT on that one.

      Joanne K

  8. HI Joanne,

    I’m assuming the preservation hold library will count as extra storage amongst your overall SPOnline data?

Leave a Reply