Building an automated SharePoint 2010 deployment system

When building a deployment system for SharePoint 2010 system, your only practical option is to use PowerShell. STSADM.EXE could in theory be used if you are only interested in SharePoint Foundation, but the moment you need SharePoint Server, things are only available in PowerShell.

SharePoint is a mature product and has its portion of quirks. Some of these come to surface when working through PowerShell. SharePoint can at times be very fond of caching information. If you keep your PowerShell session running, you may not get the result you were expecting. For instance, if you deploy a custom site definition and try to use this to create a new site using the same PowerShell session, you will be informed that SharePoint cannot find the definition. You will also experience issues when deactivating and activating features; if you have deployed updated assemblies to the GAC, the feature may actually be using an old version.

When using PowerShell and SharePoint, be prepared restart PowerShell on a regular basis. This fact is part of the driving force behind the Mystery Foundation deployment system. The deployment system is driven from a normal Command Shell; it makes it easy for you to construct new cmd files to handle your scenarios.

A copy of the deployment system and all required files can be found in the SharePoint Mystery Package.

All our deployment commands follows this pattern.

ps.cmd <Environment File> <Application> <Command> 

Environment File: the name of the ps1 file that contains environment information
Application: a string that identifies your application, you have control over this value
Command: the command you want to issue on the application, you have control over this value.

In the package you will also find BobTheBuilder.cmd that creates my demo system and ConanTheDestroyer.cmd that tears down the demo system.

File structure

Files starting with MGL are sample files and are meant to be customized by the project. Name them however you want. Files starting with SPM are part of the core deployment system and are not meant to be customized by the project.

TheServer.ps1

This file contains information for a specific environment. It contains information relevant for the farm and the various applications and services we will be using.

In my case it’s named after the server where I do the deployment from. Create one file per environment (development, testing, staging, production)

MGL.Settings.ps1

This file contains information that is relevant across all your environments.

The prefix value that is set at the farm level will be used to create the actual database name. That is why we can specify the name in a common location.

MGL.Commands.ps1

This file is where you map the various commands against the applications.  It translates the Application and Command parameters from ps.cmd into actual action. What commands you want and what you do is up to you. Copy and customize this file based on your own requirements.

The following information that is found in TheServer.cmd is important for forwarding information to this file:

SPM.Types.ps1

This is mainly a container for all the various SPM.Type.*.ps1 files. This is also where the SharePoint PowerShell module gets loaded.

SPM.Functions.ps1

This file contains a set of useful functions that are not bound to a specific type.

SPM.Type.*.ps1

There is one file per custom type. This file contains a method that returns a custom PowerShell object; it’s this object that provides all the properties for setting values and any methods that can be called.

SPM.DefaultTopology.ps1

This file creates all the objects for a default topology; all the service applications and a set of web applications. You can use this or create your own. Even if you only need a subset of this topology, you can use it. Which services and applications actually get created depends on what you do in your command file. If you create your own, stick with the same names; there are dependencies between services and applications. i.e. My Site and User Profile Service

Points of interest

Service Principle Names

The deployment system can automatically configure the SPN’s that are needed for Kerberos Authentication. This is controlled by a flag set on the farm object so that it can be turned on for a development deployment, but not in a staging or production deployment.

Databases

The system can automatically create databases at a specific location if desired.  This allows you to create the databases at a different location than the standard SQL Server location. The database folder creation is only relevant if everything is running on the same machine. i.e. developer.

In general, SharePoint will accept an empty database that it can configure, but there are some services that just refuse to configure an empty database. These will be created in the default SQL Server location.

When creating the database, we impersonate the SharePoint farm account. Some services don’t like it if dbo != SharePoint farm account. I.e. the User Profile Service

Transcript

In the command file we create a transcript file for each command.  When errors occur, the files can be sent to whoever can aid in troubleshooting.

Passwords

No passwords are stored in the deployment files. During the farm configuration process, all the required accounts are configured as Managed Accounts. When needed we read information from the Managed Account system to impersonate users.

Pending

Installing binaries and creating the farm

Currently we don’t support installing the SharePoint binaries or creating the actual farm. This deployment system assumes that the environment has been configured as far as Central Administration, but no further. The initial focus is on developers. Personally I maintain a blank SharePoint 2010 image that I use as a basis for my development; this has Central Administration already configured. There are already other scripts out there that will install the binaries and create the initial farm.

With time I plan to include this functionality as well. How soon will depend on demand.

Other stuff

There is probably a whole bunch of stuff that would be useful to include. If you have suggestions, please let me know.

Advertisements

Automating SharePoint deployments, an introduction

No matter if you are working on a 10 000 user or 100 user project, you will need to have some form of deployment system in place. It doesn’t take long before the number of settings you need configure spiral out of control. Some people like Word documents that detail all the settings with tons of wonderful screen shots. I personally hate these because they take a long time to set up and actually perform. You can also be fairly certain that at some point somebody will forget something and you spend a lot of time debugging the problem. I personally favor an automated approach; at least as much as possible. The downside is that setting up an automated deployment system takes time and money. If you are supporting 10 000 users, you may have the budget for it, but probably not for a ‘cheap’ system handling only 100 users.

The Mystery Foundation provides a lot of building blocks and a sample to help you along the way; it should drastically cut down the time to set up an automated deployment system.

Some people don’t understand the value of an automated deployment system; the guilty shall not be named. The foremost value is that it provides an indisputable documented copy of the collective knowledge that the project has when it comes to deployment. Lengthy prose can open for interpretation and unclear descriptions. A deployment system developed in PowerShell is very clear in what it does. Deployment is also something that often gets delegated to one or two people; if something should happen to them, any ‘magical’ undocumented things they may be doing also gets lost.

Automated deployment systems are also very valuable from a development perspective. If a developer knows that they can rebuild their system in a matter of hours, they are not afraid of scrapping their system and starting afresh. This uncovers any undocumented configuration changes. If a developer keeps their environment for 2 months prior to moving into test, you can be assured that some settings haven’t been document and thus not configured correctly.

Another aspect of automated deployments is that they should work across multiple environments; the information relevant to a specific environment should be separated from the rest. This ensures that you also debug your deployment system when moving between environments. When reaching staging and production, most of the bugs should have been removed and deployment into these environments shouldn’t cause many surprises.

So what are some of the attributes of a good deployment system?

Run again, and again, and again, and again...

It should be possible to run the deployment system multiple times without incurring any errors. If the database or web application already exists, it should use them and just move on.

Scoped configuration values

In any configuration you will have configuration values that are specific to the environment and values that are relevant for all environments. Examples of environment specific values include URL’s, service accounts, and database sizes. Information that can remain the same across multiple environments include timeout values, the service application names probably remain the same, as would the logical architecture.

Having a deployment system that allows you to set values for all environments or for a single environment clearly documents what is relevant for the various scopes. If you need to add a new environment, you know what values you have to come up with.

The Mystery Foundation deployment system

This deployment system is built entirely in PowerShell using the functionality provider by SharePoint 2010. Some custom Cmdlets have been developed to provide additional functionality:

  • Add entries to the host file on all servers in the farm
  • Add SPN’s so that Kerberos works
  • Create directory on all server in the farm
  • Configure Blob cache settings
  • Configure Super user accounts for the Publishing infrastructure.

Deploying Web Parts without creating an application solution

One of the things you have to ensure when deploying web parts is that the safecontrols entry is added to the web.config file. Without it, you don’t get to have any fun.

The standard way of doing this is through the manifest file of the wsp package. The challenge is that this entry makes it into an application deployed solution rather than a globally deployed solution. If you want to use this in multiple web applications, you will have to deploy it a number of times. This takes time and becomes a hassle when you have to update the solution. Depending on the changes in the solution, you may have to retract it first, then deploy it again. You then need to keep track of where it was deployed.

If you could keep it as a globally deployed solution, life would be easier and faster. Since safecontrols are only a modification to the web.config, it’s very easy to create an alternative approach.  One of the great gems of the SharePoint OM is SPWebConfigModification. This ensures that changes are applied to the web.config file on all servers for all zones. Using a web application scoped feature, you can easily apply the changes to the relevant web applications.

The challenge is that this API uses XPath a lot and can be easy to get wrong. The Mystery Foundation comes to the rescue; it contains a class called WebConfigModifications that wraps common web.config operations; one being a method for adding safecontrol entries.

Here is the actual function:

And this is how it can be used:

You can add more than one entry at a time, and when you are done, calling the Commit method will update the system.

Localizing application pages without using App_GlobalResources

ASP.NET and SharePoint both support using localized resources. It started with ASP.NET back in v2.0.

SharePoint has adopted the same type of syntax that you may use in XML files.

They may look very much alike, but they read their information from very different places. The ASP.NET version uses resx files located in the web applications \App_GlobalResources folder. The SharePoint version reads from \14\Resources folder.

So, if you want to use the standard functionality, you will have get your resx file into App_GlobalResources.

One way is to use the old trusty stsadm tool with the copyappbincontent option. Since stsadm is being replaced by the new cool kid on the block going under the handle PowerShell, this option may not fly.

Another way is to get SharePoint to deploy it there for you. By modifying the manifest.xml file you can get SharePoint to deploy your resource files in the correct location. If you want your resx file both under the web application and SharePoint root folder, you will have to get your hands dirty with some XML. Another side effect is that your wsp suddenly becomes application deployed rather than globally deployed.

If you have several web applications, the files have to be updated in all locations. If one file isn’t updated, you may have a bug that shows up in some places and not others; not fun to debug.

The third option is to leverage the framework ASP.NET has given us. The syntax ASP.NET uses is processed by an expression builder; it’s an extensible framework that SharePoint and we can use. The Mystery Foundation contains an implementation that has the same behavior as the SharePoint XML files. By activating the SPM.AppInfrastructure feature on your web application, it becomes available for you to use.

In the web.config file a new entry is added:

It’s then possible to use it like this in an aspx page:



Using this approach, it’s easy to use resources in your web pages and easy to deploy and update the system.

Magical Mystery Tour Overview

The Magical Mystery tour is a single Visual Studio Solution that contains a number of Visual Studio projects. The exact number will vary over time, but the current configuration is showed in the screen shot below. From this, we can see that there are 2 mystery projects, the rest being the magical ones.

The Mystery.SharePoint.Foundation contains all the basic stuff. All the functionality in this project will be available on SharePoint Foundation. The Mystery.SharePoint.Server project contains stuff that depends on SharePoint Server.  There is also a magical project for the common types of applications that SharePoint is used for. The idea behind this breakdown is to make updating the system easier. If you make changes to your search or publishing presentation, this should not affect your collaboration or record center applications.

At this point in time, a lot of the projects don’t contain anything. They purely exist so that I have space to grow, and it illustrates how one can structure more complex SharePoint systems.

Throughout the system you may see two naming conventions. SPM = SharePoint Mystery. MGL = Magical.

I will try to keep compatibility for future releases of the SPM, so if you modify code here, all bets are off. If you have improvements you think I should include, I’d love to hear about them.

The Magical Mystery Tour begins

If you are new to SharePoint, you have undoubtedly stared down the tunnel and wondering if the light shining in your eyes is hope or a freight train coming at you. This project is all about sharing my take on the SharePoint thing.

I’ve been working with SharePoint for a number of years now, and a number of things keep cropping up on multiple projects.

  • Deploying and setting up SharePoint installations
  • Site collection provisioning and maintenance
  • All sorts of other bits and pieces that makes SharePoint development easier.

The purpose is to create a package that contains useful additions to SharePoint that makes your life easier.

You can either:

  • Use the package as is
  • Copy the pieces of code you want
  • Or, totally ignore the whole thing. If you decide to do this, I would like to hear why.

It can be downloaded from CodePlex

Now an explanation of the name may be in order. The base part of the project is called the SharePoint Mystery; this contains stuff that is generic across multiple projects. The sample project is called Magical. This does in no way insinuate that I’m a magician. It’s simple the fact that when you combine Magical and Mystery, you get the part of the title of a Beatles song. Since I’ll be showing you around, it will be a Tour. Simple as that.

In the weeks and months ahead, various blog posts will describe certain parts of the code. If there is an area you are interested in that I haven’t written anything about, send me a message. If you find errors or have suggestions on how things can be improved, use the issue tracker on CodePlex.