Multiple Libraries, Folders or Metadata?

Blog Post: 5 minute read

The other day I was part of a conversation on the subject of storing a group of categorized documents in SharePoint. The gist of the question was around wanting to know if it was best to create multiple document libraries (1 for each category), create 1 document library with multiple category folders within it or use metadata rather than folders. This is an age-old conversation that has gone on thru the years in the SharePoint world. 🙂

“Category” in this post means any kind of high-level criteria or group you’re wanting to organize your documents by. Examples of this could be project name, document type, etc.

As most people familiar with SharePoint will tell you, all options will work at times however one may be better suited than the others in any given situation. Your decision will ultimately be based on your unique business requirements and these six factors:

Circle12

  1. Permissions
  2. Volume
  3. Retention
  4. Audience
  5. Requirement to change category
  6. Search

The 3 options we will consider while evaluating each factor are:

  • Option 1: create  1 document library per category
  • Option 2: create 1 document library with 1 folder for each category
  • Option 3: use a required metadata column for each category

[UPDATE March 22, 2017] I neglected to mention document sets in this post. I usually describe these to users as “smart folders” since the folder can itself have metadata. It is an excellent choice for grouping documents together in a folder with shared metadata. Examples of this are contract files, project files, job competitions, etc.


Factor 1: Permissions

What are the security requirements for the content? If there are different permissions per category your options are:

  1. Option 1: Create 1 document library per category. Set the permission at the library level. Good choice.
  2. Option 2: Create 1 document library with multiple category folders within it. Break permission inheritance from the library level and set it at the folder level. Ok choice.
  3. Option 3: You can not allow for permission differences between different metadata values so this is not an option.

Circle4My opinion: Both options 1 and 2 will work, however I lean toward trying to set permission at the library level (highest level) in SharePoint wherever possible. This is simply due to the negative performance hit whenever permission inheritance is broken as well as ease of permission administration.

Conclusion: Permissions alone will likely not dictate which option you choose. Your choice will often come down to other factors.


Factor 2: Volume

You should be aware of the list view threshold limit when working with lists and libraries in SharePoint. Know the rough estimate of the maximum number of documents that can be in any one category at any point in time. This will indicate how many documents you may want to display when a user is viewing the library. If you will exceed the threshold when displaying the documents this is a problem.

  1. Option 1: If the volume of documents in any category will exceed the limit, you can choose to separate them into separate libraries and then also have a top-level folder level to further break down the number of documents you will want to display at one time. This is a good choice.
  2. Option 2: If the volume of documents exceeds the threshold limit you may want to use a top-level folder structure for the category and an additional folder level under that to further break down the number of documents returned in any view.
  3. Option 3: If the volume of documents exceeds the threshold limit, then metadata alone is not sufficient. You may want to use some of the metadata values as folders instead in order to stay under the limit.

Circle10My opinion: If it exceeds a couple thousand documents, you will likely want to have at least a top-level folder structure to group the documents. End-users will receive a threshold error if any view they use exceeds  5000 items. (SharePoint Server 2016 can go beyond this limit) You will run into this, for example, if you try to view documents without folders and the # of items returned exceeds 5000 items. I would carefully design your library around the limit and additionally ensure indexes are created on metadata you will be filtering your views on.

Note that in SharePoint Server 2016 and SharePoint Online, an auto-indexing feature will automatically create a column index once you reach 2500 items in a list/library and it is used to filter a view.

For large libraries, it is usually not feasible for users to navigate the library to find their document particularly if you have a lot of folders. Instead, encourage users to leverage search. A good strategy is to have some metadata  defined in the library and use them as refiners on a search page in the Enterprise Search Center.

You may also want to consider using the Content Organizer feature in SharePoint for large libraries. This can automatically create folders based on metadata and route documents into them. It can be configured to automatically create a new folder after, for example, every 2000 documents to ensure performance is kept in-check. Be aware there are limitations to this feature particularly for required metadata on a library as it will leave documents checked out if you submit multiple documents at a time to the drop-off library.

Conclusion: Have a good handle on the expected volume of documents in your library. Based on this you may want to create separate libraries in order to spread out the volume and/or create a simple folder structure within it.


Factor 3: Retention

You can set retention at numerous levels. (site, content type, library, folder)

  1. Option 1: You can set retention at the library level. All content within the library will fall under the same retention rules.
  2. Option 2: You can set retention at a  specific folder level. All content within the folder will then fall under the same retention rules
  3. Option 3: You cannot set retention based on metadata directly. Out-of-the-box a date field is required to set up a retention rule for either document, folder, or content type retention. You can use something other than date with custom code.

Circle15My opinion: One important limitation of folder-based retention is if you allow end-users to create folders, then you will have to continually set retention for each new folder after it is created. (it will default to the document library retention if you don’t) If you require more granular control over different retention options you should use content types in your document library and set retention based on that. This allows the same retention to be applied across all libraries and folders using that content type which is definitely a more consistent, maintainable approach.

Conclusion: It really depends on the complexity of your retention requirements. Wherever possible I prefer to set retention at the content type level. This allows you to have different types of content across a site collection/site/library/folder each with different retention options. If you have simpler retention requirements (i.e. everything in the document library/folder has a retention of 10 years) then setting it at the document/folder level is an option and is sometimes sufficient.


Factor 4: Audience

In my opinion, this is likely one of the most important factors to consider.

Does this library have a few contributors and many readers? If this is the case, then you should only use folders for the benefit of the contributors. Let them come up with a simple folder structure  (2 at most) they can all agree on and additionally tag the content with metadata. This will allow the flexibility of building search pages with metadata-based refiners. The majority readers will then use the Enterprise Search Center to find their content.

Circle2Does this library have a lot of contributors and very few readers? If this is the case, you will want to be very careful with the use of folders as there are likely as many ways to structure them as their are users contributing to the content. At times, folders are a good choice for a top level organization strategy within a list or library. You may want to consider setting up folders ahead of time and then turning off folder creation at the library level in order to eliminate abuse of folders in a library (although you will likely get kick-back from the users on this). If you choose to use metadata, spend the time to decide how users will want to filter/find their content and only do the absolute minimum metadata required to accomplish this.

Conclusion: Too much of either folders or metadata can be a bad design decision. I have seen very effective solutions using a judicious combination of both. This sets you up for an effective search experience while still providing helpful views for contributors. This is where you may also want to use the column default values feature to automatically set metadata in a folder based on the folder name.


 Factor 5: Requirement to Move Content from 1 Category to another

Often there is a requirement to move content from 1 category to another and there are currently several ways to do this. I have written a blog post describing 10 ways to move content from 1 library to another.

  • Option 1: Moving between document libraries will cause you to lose version history. It is also difficult to do as (currently) the capability to do this thru the UI is not there.
  • Option 2: Moving between folders in a document library can be done thru the SharePoint UI and version history is maintained. The new ‘Move to’ functionality in the modern document library view allows you to easily move from 1 folder to another. This is super useful.
  • Option 3: Changing the metadata value is simple at any time.

Circle6My opinion: One big drawback of moving from one library to another is the loss of version history. If this is a requirement, then you should not store your content in separate libraries. With the new ‘Move to’ capability introduced in the Modern Document library this allows you to move between folders within a library but not between libraries. Hopefully this capability will come in the future although it still may not resolve the version history issue.

Conclusion: If it is a requirement to retain version history then you should stay within one library for all of your categories and use either folders or metadata.


Factor 6: Search

Allowing users to use search to find their document is a value-add particularly if there is a lot of documents in the library.

  • Option 1: If documents are housed across multiple libraries, users will have to know which document library the document is in to find it so it’s best for users to use the Enterprise Search Center instead.
  • Option 2: If your documents are housed in multiple folders in one library, users will have the option of using the search box at the top of the library level to find it or going to the Enterprise Search Center to search from there.
  • Option 3: if you are using metadata in a library, users will have the option of using the search box at the top of the library level or going to the Enterprise Search Center to search from there. Metadata is key for a great search experience!

Users can also use the Delve app in O365 to get back to recent documents they’ve worked on and to search for documents. It is a great productivity tool however it does not replace the need for a customized Enterprise Search Center in your environment with targeted content, customized refiners and display templates.

Circle8Conclusion: Users should use the Enterprise Search Center (or Delve) if they don’t know where their document is located. This means you should spend some time tagging your content with metadata to enhance the Enterprise Search experience. Conversely, if a user knows the library a document is located in, they can navigate directly to it and use the search box at the top of the library.


Summary

Like most things in SharePoint there is no clear-cut answer to the question whether you should choose multiple document libraries, 1 document library with multiple folders, metadata or a combination of all three. Hopefully if you consider the factors discussed in this post you will be armed with the knowledge to make the right decision for your specific business scenario.

Thanks for reading.

-JCK

10 comments

  1. Hello Joanne, its really appreciate of you sharing on the SharePoint metadata, Lib & folders.
    I am actually thinking about all these recently as I have appointed to design a site for my team which will have around 100 users and more than 7 teams. I am thinking to create few subsites to represent each teams and create few libraries and folders in each subsites. As requested by teams, there are different permission for users to access libraries and folders. I have few questions hope u can give me some insight. And I am on SharePoint 2016 now.

    (1) if I use Metadata in a library (number of folders are created in this library but different permission has been setup for each folder), is that means user can search the files in the folders which they have access right only?

    except those super users which they have all folders access then they are able to search all the files by Metadata?

    (2) what is the different between “search box” on the top right hand site and “Enterprise search center”? where is the Enterprise search center in SharePoint?
    I heard the metadata has limitation which can be created in a library or folder only, is this true? I am thinking if I create a metadata in my HOME site, can it search all files from all sub-sites, libraries and folders..(it seems too complicated or seems not possible?!)
    is that the reason you are suggested to use “search box” or “Enterprise search center”

    Thank You very much!
    Rgds, Annie

    1. Hi Annie,

      Question 1: You are correct, if you have permission-trimmed your folders and a user searches the library they will only have search results returned to them from the folders they have access to.
      Question 2: The Enterprise Search Center is a site template in SharePoint. You can build search pages in there that are scoped outside of the current site and site collection you’re currently in. If you use the top right hand search box, it will only search within the current site (or document library if you select that option). You can adjust the search settings so if you use the top right hand search box it will take you to the Enterprise search center.
      Question 3: If you intend on re-using metadata, it is a good idea to define it at your site level and re-use it across all of your libraries within your site. This will give you some consistency and will allow that piece of metadata to be used as part of a search query. If a piece of metadata is defined as a site column, it will have a managed property created for it which means it can be used in search. If you only create a metadata column at the library level, it will not have a managed property generated for it. There are many other factors to consider when building out search solutions, but that is a big one. Also … be careful about building sub-sites in your structure. It is best to keep your Information Architecture as flat as possible.

      Hope this helps.
      Joanne K

      1. Dear Joanne

        Thanks for your prompt response.
        But some points I don’t very understand, hope you can explain abit more or can give me an examples,

        Q2: When use “Search box” I can only see “search this site only” option, where I can set up “document library” option as well?
        Also, let’s say is it possible to use metadata columns to search the result in “Search box”?
        example: if I have defined a metadata column Deliverable Type with values “XXX submission” or “YYY submission”
        Then in “Search Box” if I input XXX submission, will all files come out if it has this column tagged?

        Q3: Our site will be used by different teams and many restriction on access for each folder. Also users have different request in building metadata columns in each library. Of course I am trying to maintain same metadata columns for them but 1 or 2 columns may be different in each library due to team has different requirements on their files.
        from your comments it sounds that we can also build metadata columns in HOME site, so it can search from sub-sites and libraries as well. But do you have example or how to set it up? I am not quite understand how it works. And you have mentioned the Site columns but what is this, sorry I am abit confuse ..

        Thank You
        Rgds, Annie

  2. Love this article. And as a SP consultant, I have been trying to explain this to users for 10 years (although not as clearly as you have done in this article.).

    There is a another factor you could consider – change of document status. Consider a set of documents that goes from Draft/Review (a small team of contributors to view/edit) > Approved (all the business to view) > Archived/Obsolete (no one to view). I have seen this scenario many times and people struggle with it.

    Libraries = terrible idea. Although permissions are easy to set on the library, moving documents between libraries is (currently) an awful experience.

    Metadata = a column called “Status” helps a lot to change the status, and group documents by status. But if it were a requirements to “hide” the Archived documents, metadata alone cannot solve this problem because we cannot use the Status column to set different permissions.

    Folders = an OK idea because we can easily move documents from one folder to another. We can also set permissions on each folder.

    (Note: a fourth option = Record Declaration and Retention could be used to solve this problem but this requires a whole separate setup, and many clients are not sophisticated enough for this technical solution.)

    Here is a scenario where a metadata column would be advantageous to a certain audience, but folders may be required to hide the old obsolete documents.

    A very, very common thing I also see is working documents (Word) in one library and the final approved version (PDF) in another library, but this is just soo bad (from a Document Management perspective), it can wait for another article.

    Cheers!

    1. Hi Mike,
      I agree – great example! When I get a few minutes, I will update the post with your example. This is a very common scenario and I agree with your assessment.
      Joanne K

Leave a Reply