Site Collection Provisioning: meta-data

For some types of sites it makes sense to have additional meta-data. If you have a project site, the presence of a project number or charge code would allow the system to fetch information from other systems and display it to the user. It’s also important to enforce this no matter how the site was created. If the site was created from a third party tool, the first time a user accesses the site, they should be prompted for filling in the required information.

Our way of solving this involves a number of components.

The first step is to create a meta-data page where users can edit whatever is relevant for their site. Different types of sites will have different types of data, so a flexible approach is needed.

With the Mystery Foundation you get a meta-data page. This is a simple application page with space for a delegate control.

The information about the owner is generic for all types of sites, whilst the project information is specific for a type of site. You can use the delegate control to add additional information to this page.

The control that you implement must implement the interface Mystery.SharePoint.ISiteMetadataControl. This will handle sending information between the page and the back-end storage.

The next step for handling meta-data is to capture it somewhere.

We have a feature, SPM.MetadataRequired, that sets the provider used to capture the data. This feature will also add a site action link under site settings that takes you to the meta-data page. The object used as the provider must inherit from Mystery.SharePoint.SiteMetadata. Your custom type will be passed to the ISiteMetadataControl methods.

Access to your meta-data can be achieved using the following method. Specify the type you want returned; it can either be your inherited type or a base type.

Now that you have wired up the meta-data handling, you  have to ensure that any required data is filled in. Remember that the site may have been created in many different ways; central admin, PowerShell, custom site collection provision UI, third party tool, etc.

The only reliable way to handle all these scenarios is when the user accesses the site for the first time. The most elegant way is to use a delegate control that redirects you to the meta-data page. Ensure that the  SPM.MetadataRedirect feature is activated as part of your web template.

When the user has filled in the required information, the feature will be deactivated, so on subsequent visits, things work as normal.  Using a delegate control only adds an overhead until information has been added. If you add some logic to the master page, the overhead will be incurred on every page visit to the site.


Site collection provisioning

When you have reached the decision to use multiple site collections, you need to think about a number of things.

  • Who should be able to provision site collections?
  • Should only a predefined set of templates be available in the web application?
  • From where will provisioning take  place? Web UI, PowerShell, third party tools?
  • How do you manage users across all these site collections?
  • Who should be the site collection administrators of the new site collection?
  • What type of approval system should be in place?
  • How does one determine when a site collection is no longer in use so that the space can be reclaimed?
  • Does the site collection require any additional meta-data?
  • What Quota should be applied to the site collection?

The Mystery foundation contains functionality that helps solve a number of these issues. I haven’t solved all of the them yet, but I’m working on it.

The current solution consists of the following components:

  • A custom application page that allows users to provision site collections
  • An object to store various configuration data:  Mystery.SharePoint.SiteProvisioningSettings.
  • A timer job that handles various site administration tasks:  Mystery.SharePoint.SiteManagementJob

Custom page:

Standard page:

Deciding what data to capture

Comparing these two pages, they are very similar in what data they capture; this is by design. The values captured here is basically the information needed to create a site no matter how you do it; Web UI, PowerShell, Object model. When you operate in a UI world, it’s easy to go overboard and ask for all sorts of additional information. It works great as long as you control the  UI, but the day you need to create 100 sites from a third party tool, you get problems. Better to plan for it from the start because you never know what is coming around the next corner.

Limiting the number of templates

One of the major difference between the standard and custom page are the number of available templates. You probably don’t want to allow everybody to create any type of site.  The standard page has no way of limiting the selection, so you will have to roll out your own or use the one we provide.

One of the values stored in the SiteProvisionSettings is a list over what templates should be made available. A custom developed web template picker uses this information to present the reduced selection.  The SiteProvisionSettings object can be retrieved and updated using the Get-SPMSiteProvisioningSettings PowerShell Cmdlet.

Limiting the user that may provision sites

One of the other settings in the SiteProvisionSettings is a list of Authentication providers and the active authentication provider. This is an extensible framework, so you can create your own. Just implement Mystery.SharePoint.ISiteProvisioningAuthenticationProvider and register it using PowerShell. The package contains two authentication providers; one that allows anybody to create site collections, the other only allows members of the root site collection to provision them.

User management

When you have a system with multiple site collections, handling users becomes an issue.  Most companies are all about sharing information. The moment you create a new site collection, you create a new security container that by default doesn’t allow everybody access. This goes against the information sharing aspect. In order to overcome this, we have included a site scoped feature SPM.SynchronizeVisitors. If this feature is activated, all the members of the visitor group at the root site collection will be copied to the visitor group of the provisioned site collection. The timer job will make sure that it’s up to date every time it runs. If you have certain types of sites that shouldn’t be available to everyone or want to turn it off, just deactivate the feature and you can control your visitor group manually.

Another area the can cause issues is the site collection administrators. These have to be added individually to a site collection; you cannot specify an active directory group. Certain people should probably have access to all site collection. Keeping this updated across 100s of site collections is a challenge. The site collection feature SPM.SynchronizeAdmin will behave in same way as the visitor synchronization. Activate it based on your needs.

When a user requests a new site collection, do you want them to become a site collection administrator? With this comes a lot of power; are they ready for it? If you want to give people this level of power, make sure that the site collection feature  SPM.OwnerRemainsAdmin remains activated. If this is not activated, the user that requested the site collection will no longer remain and administrator, but rather become a normal owner.

Quota assignment

Major corporations will require a set of different site types. i.e. Projects and Departments. In order to prevent information overflow, they may want to limit the amount of data in each site collection.  The storage needs for a project is probably very different from a department, so it’s not unlikely that they will use different quotas. How does one ensure a different quota is applied for different site collections?

One of the things to be aware of is that you can only change the quota a site collection uses if you are running as a farm administrator. This means that we cannot update our site collection immediately. We have to tell the system what quota we should be using, and then have the timer job set the actual quota. This allows the site to have a free quota for a period of time, but chances of them exceeding it before it is enforced is rather slim.

In order to leverage the quota management, make sure that you activate the SPM.QuotaAdmin feature as part of your web template.

The value specified as the default template will be assigned by the timer job. This feature will also give owners an overview over how much space their site is consuming. A notification bar will also be presented when the storage space approaches critical limits.

(in this sample I adjusted the quota after adding content, hence the 112% usage)

Additional meta-data

Some types of site collections may require additional meta-data.  I.e. a project site may need a project number so that it can use this to fetch data and display data from additional systems. SharePoint doesn’t natively support site collection meta-data, so we have to create our own system. A more detailed description on how one can create such a system will be left for a future blog post.

Building an automated SharePoint 2010 deployment system

When building a deployment system for SharePoint 2010 system, your only practical option is to use PowerShell. STSADM.EXE could in theory be used if you are only interested in SharePoint Foundation, but the moment you need SharePoint Server, things are only available in PowerShell.

SharePoint is a mature product and has its portion of quirks. Some of these come to surface when working through PowerShell. SharePoint can at times be very fond of caching information. If you keep your PowerShell session running, you may not get the result you were expecting. For instance, if you deploy a custom site definition and try to use this to create a new site using the same PowerShell session, you will be informed that SharePoint cannot find the definition. You will also experience issues when deactivating and activating features; if you have deployed updated assemblies to the GAC, the feature may actually be using an old version.

When using PowerShell and SharePoint, be prepared restart PowerShell on a regular basis. This fact is part of the driving force behind the Mystery Foundation deployment system. The deployment system is driven from a normal Command Shell; it makes it easy for you to construct new cmd files to handle your scenarios.

A copy of the deployment system and all required files can be found in the SharePoint Mystery Package.

All our deployment commands follows this pattern.

ps.cmd <Environment File> <Application> <Command> 

Environment File: the name of the ps1 file that contains environment information
Application: a string that identifies your application, you have control over this value
Command: the command you want to issue on the application, you have control over this value.

In the package you will also find BobTheBuilder.cmd that creates my demo system and ConanTheDestroyer.cmd that tears down the demo system.

File structure

Files starting with MGL are sample files and are meant to be customized by the project. Name them however you want. Files starting with SPM are part of the core deployment system and are not meant to be customized by the project.


This file contains information for a specific environment. It contains information relevant for the farm and the various applications and services we will be using.

In my case it’s named after the server where I do the deployment from. Create one file per environment (development, testing, staging, production)


This file contains information that is relevant across all your environments.

The prefix value that is set at the farm level will be used to create the actual database name. That is why we can specify the name in a common location.


This file is where you map the various commands against the applications.  It translates the Application and Command parameters from ps.cmd into actual action. What commands you want and what you do is up to you. Copy and customize this file based on your own requirements.

The following information that is found in TheServer.cmd is important for forwarding information to this file:


This is mainly a container for all the various SPM.Type.*.ps1 files. This is also where the SharePoint PowerShell module gets loaded.


This file contains a set of useful functions that are not bound to a specific type.


There is one file per custom type. This file contains a method that returns a custom PowerShell object; it’s this object that provides all the properties for setting values and any methods that can be called.


This file creates all the objects for a default topology; all the service applications and a set of web applications. You can use this or create your own. Even if you only need a subset of this topology, you can use it. Which services and applications actually get created depends on what you do in your command file. If you create your own, stick with the same names; there are dependencies between services and applications. i.e. My Site and User Profile Service

Points of interest

Service Principle Names

The deployment system can automatically configure the SPN’s that are needed for Kerberos Authentication. This is controlled by a flag set on the farm object so that it can be turned on for a development deployment, but not in a staging or production deployment.


The system can automatically create databases at a specific location if desired.  This allows you to create the databases at a different location than the standard SQL Server location. The database folder creation is only relevant if everything is running on the same machine. i.e. developer.

In general, SharePoint will accept an empty database that it can configure, but there are some services that just refuse to configure an empty database. These will be created in the default SQL Server location.

When creating the database, we impersonate the SharePoint farm account. Some services don’t like it if dbo != SharePoint farm account. I.e. the User Profile Service


In the command file we create a transcript file for each command.  When errors occur, the files can be sent to whoever can aid in troubleshooting.


No passwords are stored in the deployment files. During the farm configuration process, all the required accounts are configured as Managed Accounts. When needed we read information from the Managed Account system to impersonate users.


Installing binaries and creating the farm

Currently we don’t support installing the SharePoint binaries or creating the actual farm. This deployment system assumes that the environment has been configured as far as Central Administration, but no further. The initial focus is on developers. Personally I maintain a blank SharePoint 2010 image that I use as a basis for my development; this has Central Administration already configured. There are already other scripts out there that will install the binaries and create the initial farm.

With time I plan to include this functionality as well. How soon will depend on demand.

Other stuff

There is probably a whole bunch of stuff that would be useful to include. If you have suggestions, please let me know.

Deploying Web Parts without creating an application solution

One of the things you have to ensure when deploying web parts is that the safecontrols entry is added to the web.config file. Without it, you don’t get to have any fun.

The standard way of doing this is through the manifest file of the wsp package. The challenge is that this entry makes it into an application deployed solution rather than a globally deployed solution. If you want to use this in multiple web applications, you will have to deploy it a number of times. This takes time and becomes a hassle when you have to update the solution. Depending on the changes in the solution, you may have to retract it first, then deploy it again. You then need to keep track of where it was deployed.

If you could keep it as a globally deployed solution, life would be easier and faster. Since safecontrols are only a modification to the web.config, it’s very easy to create an alternative approach.  One of the great gems of the SharePoint OM is SPWebConfigModification. This ensures that changes are applied to the web.config file on all servers for all zones. Using a web application scoped feature, you can easily apply the changes to the relevant web applications.

The challenge is that this API uses XPath a lot and can be easy to get wrong. The Mystery Foundation comes to the rescue; it contains a class called WebConfigModifications that wraps common web.config operations; one being a method for adding safecontrol entries.

Here is the actual function:

And this is how it can be used:

You can add more than one entry at a time, and when you are done, calling the Commit method will update the system.

Localizing application pages without using App_GlobalResources

ASP.NET and SharePoint both support using localized resources. It started with ASP.NET back in v2.0.

SharePoint has adopted the same type of syntax that you may use in XML files.

They may look very much alike, but they read their information from very different places. The ASP.NET version uses resx files located in the web applications \App_GlobalResources folder. The SharePoint version reads from \14\Resources folder.

So, if you want to use the standard functionality, you will have get your resx file into App_GlobalResources.

One way is to use the old trusty stsadm tool with the copyappbincontent option. Since stsadm is being replaced by the new cool kid on the block going under the handle PowerShell, this option may not fly.

Another way is to get SharePoint to deploy it there for you. By modifying the manifest.xml file you can get SharePoint to deploy your resource files in the correct location. If you want your resx file both under the web application and SharePoint root folder, you will have to get your hands dirty with some XML. Another side effect is that your wsp suddenly becomes application deployed rather than globally deployed.

If you have several web applications, the files have to be updated in all locations. If one file isn’t updated, you may have a bug that shows up in some places and not others; not fun to debug.

The third option is to leverage the framework ASP.NET has given us. The syntax ASP.NET uses is processed by an expression builder; it’s an extensible framework that SharePoint and we can use. The Mystery Foundation contains an implementation that has the same behavior as the SharePoint XML files. By activating the SPM.AppInfrastructure feature on your web application, it becomes available for you to use.

In the web.config file a new entry is added:

It’s then possible to use it like this in an aspx page:

Using this approach, it’s easy to use resources in your web pages and easy to deploy and update the system.

Magical Mystery Tour Overview

The Magical Mystery tour is a single Visual Studio Solution that contains a number of Visual Studio projects. The exact number will vary over time, but the current configuration is showed in the screen shot below. From this, we can see that there are 2 mystery projects, the rest being the magical ones.

The Mystery.SharePoint.Foundation contains all the basic stuff. All the functionality in this project will be available on SharePoint Foundation. The Mystery.SharePoint.Server project contains stuff that depends on SharePoint Server.  There is also a magical project for the common types of applications that SharePoint is used for. The idea behind this breakdown is to make updating the system easier. If you make changes to your search or publishing presentation, this should not affect your collaboration or record center applications.

At this point in time, a lot of the projects don’t contain anything. They purely exist so that I have space to grow, and it illustrates how one can structure more complex SharePoint systems.

Throughout the system you may see two naming conventions. SPM = SharePoint Mystery. MGL = Magical.

I will try to keep compatibility for future releases of the SPM, so if you modify code here, all bets are off. If you have improvements you think I should include, I’d love to hear about them.

The Magical Mystery Tour begins

If you are new to SharePoint, you have undoubtedly stared down the tunnel and wondering if the light shining in your eyes is hope or a freight train coming at you. This project is all about sharing my take on the SharePoint thing.

I’ve been working with SharePoint for a number of years now, and a number of things keep cropping up on multiple projects.

  • Deploying and setting up SharePoint installations
  • Site collection provisioning and maintenance
  • All sorts of other bits and pieces that makes SharePoint development easier.

The purpose is to create a package that contains useful additions to SharePoint that makes your life easier.

You can either:

  • Use the package as is
  • Copy the pieces of code you want
  • Or, totally ignore the whole thing. If you decide to do this, I would like to hear why.

It can be downloaded from CodePlex

Now an explanation of the name may be in order. The base part of the project is called the SharePoint Mystery; this contains stuff that is generic across multiple projects. The sample project is called Magical. This does in no way insinuate that I’m a magician. It’s simple the fact that when you combine Magical and Mystery, you get the part of the title of a Beatles song. Since I’ll be showing you around, it will be a Tour. Simple as that.

In the weeks and months ahead, various blog posts will describe certain parts of the code. If there is an area you are interested in that I haven’t written anything about, send me a message. If you find errors or have suggestions on how things can be improved, use the issue tracker on CodePlex.