Migrating large volumes of enterprise data to SharePoint
Erik Raita Erik Raita
Managing Director, Co-founder

SharePoint is going strong, no doubt. One can even see signs of hype, as it not only conquers Lotus Notes as ECM platform, but has also became a real alternative for the specialized document management systems like Documentum or Hummingbird. Surprisingly, organizations have lately started moving their document management to SharePoint. Obviously, many useful features of a professional DM system are lost after this transition, since SharePoint does not provide a full set of DM tools. However, even a less sophisticated user experience of SharePoint in compare to, say, Documentum is not preventing that trend. Microsoft is doing good job in winning a market share not only from IBM on ECM market, but also from EMC and Open Text on a DM market. Why does this happen? There are few reasons, such as:

  • SharePoint is used by 78% of Fortune 500 companies. Market share has increased more than 40% since 2006 and still expanding. Obviously, this also means decreasing market share of traditional DMS providers making it more difficult to find high quality resources for the maintenance of legacy systems.
  • Total cost of ownership for SharePoint is app. three times lower if compared with Documentum. In recent years Microsoft has shaped the very competitive ecosystem of qualified service providers for SharePoint solutions.
  • Attractive combination of Enterprise Content Management / Enterprise Document Management / Business Intelligence / Collaboration / Social networking functionalities in one common platform, which is being permanently improved since the first release in 2001.
  • Seamless integration with MS Office applications; Office Web Applications, popular open networks.
  • Quick integration with almost any backend systems via open standardized Business Connectivity Services framework and available free add-ons (for example, integration with Documentum in ~30 minutes, http://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=15493)
  • Almost unlimited scalability; standardized extensible framework of powerful search connectors that can provide searching documents in almost any type of media (file shares, internal and external web sites, remote blobs etc.)
  • Native support of Directory Services (Active Directory) and any possible type of authentication (NTLM, Kerberos, OXML, Claims etc.)
  • Support of the world-class software vendor that quickly reacts on changing market needs and provides clear roadmaps of new product releases for 3-4 years ahead.
  • ASP.NET development platform that provides flexible programming in several industrial languages C++ / C# / VB plus easy scripting with Powershell.

Let’s now assume you have already made a decision in favor of Microsoft, and SharePoint is going to be your new consolidated ECM and DM platform. Your next challenge is bulk data migration. Training users and winning their opposition is actually even bigger challenge, but let’s concentrate here on technical issues. The dream of any IT manager is the migration of whole content as quickly as possible. Quickly means with as little errors and incompatibilities as possible. And, of course, quickly must mean cost-effectively. Speaking about expenses, it’s worth to notice that costs of any data migration usually consist of two parts, the direct cost of the software application that provides migration (“migration tool”), as well as direct and indirect costs of a data migration itself and time reserved for fixing migration errors that occur due to incompatible data formats, network problems, etc. Eventually, the more sophisticated tool is used the less errors occur. So the selection of a proper migration tool is a key issue for the successful outcome. And here one usually has to decide whether to use a commercial product or to rely on a custom made solution. Unfortunately, there’s no painless migration, no matter what was the decision. The point is just to minimize the pain.

There are several commercial migration tools on the market. The most known are Visionet’s VisiMigrate (Lotus Notes to SharePoint), Avepoint’s DocAve (Lotus Notes, Documentum, Vignette to SharePoint), Tzunami Inc.’s Tzunami Deployer (Lotus Notes, Documentum, Hummingbird, Aqualogic to SharePoint), and Verinon's Easy2Share (Lotus Notes, Documentum to SharePoint). Using a commercial tool seems to be the safest way for obvious reasons. You purchase the license, allow the tool perform data migration, and get supported by the vendor if something goes wrong. However, in practice things are not that simple. First, the license cost is often dependent on the volume of data, and in case of large data storages may become unreasonably expensive (even tens of thousands euro … only for a tool). Moreover, you don’t always even know the exact volume of your data, which makes it difficult to negotiate a proper price. Second, each migration is unique, and standard tools can serve the specific needs only to certain extent. In one of our recent cases, a well-known commercial tool was first chosen by the customer after the preliminary investigation of its features and capabilities. However, the amount of unexpected mismatches that occurred during the test migration was so large that the customer has decided to use a custom tool in the end. Finally, perhaps the biggest problem of using commercial tools steps into the picture when the bulk migration is done and one has to start fixing errors, which are practically inevitable. Getting seamless and quick support from the tool vendor turns out to be problematic, as they are typically far away and in very different time zone. That’s assuming they are otherwise eager to help, which is, unfortunately, not always the case.

To summarize, using standard tools is reasonable when both output and input systems have no customizations or non-standard features, and when the amount of data to be migrated is moderate, say around 30 GB to justify the price. Contrary, using a custom tool makes sense in case of large volumes of data and when custom or unique features are employed in the current system or specific requirements should be applied on the SharePoint side. Typical examples of such features and requirements include:

  • Document versions with several different names or file extensions for the same document in the source system;
  • Presence of one document in more than one folder or workspace in the source system;
  • Decision to migrate content into more useful folder structures in SharePoint when folder names should be based on specific combination of metadata fields bound to source documents;
  • Use of version trees, non-numeric version labels, branches like 1.2, 1.2.1, 1.2.2,,, 1.3 in Documentum that usually require non-standard handling techniques;
  • Need to use precise version numbers for target folders in SharePoint (yes, this is possible);
  • Requirement to add specific shortcuts to folders by location independent IDs;
  • Migration of specific areas of source system in specific orders with automated post-migration actions;
  • Automated migration to multiple target sites and libraries without annoyingly long manual mapping of attributes proposed by most of commercial tools. Or course, custom tools are common solutions that should work in common way everywhere; flexibility is secondary there.

Based on our practice we can state that custom migrations are usually more expensive than the simplest cases automated with commercial tools, but much less expensive in case of complex migrations of large data volumes that typically require tight control over the process and advanced flexibility in handling specific cases.  By the way, custom in our case does not mean "developed from the scratch", but rather based on the proven export-import application developed by NED, which serves as a core for the migration tool being flexibly adjustable to match specific or non-standard requirements.

According to our experience with multiple large cross-platform data migrations (Documentum to SharePoint, Lotus Notes to SharePoint, Hummingbird to SharePoint, DocMan to SharePoint, SharePoint 2003 to SharePoint 2010), we’d like to make the following list of recommendations and checkpoints.

  • Despite of vendors’ promises, there are no seamless migrations. So be prepared for technical challenges.
  • Carefully define exact output and input data formats, no matter whether you use commercial or custom tool. The more time you dedicate to the planning the less time and money you spend on fixing errors.
  • Carefully evaluate the applicability of a commercial tool for your specific needs. You better spend a limited budget on this evaluation performed by an experienced consultant than spend much more money later on fixing unexpected errors and mismatches. Frustration of end users with your new DM system from the very beginning is definitely not what you expect.
  • Hire/contract only experienced consultants with proven expertise. The role of the real practical experience is highly emphasized in case of complex cross-platform migrations. Applying “normal” selection criteria like turnover or number of personnel for the service providers is not needed here, since projects are typically very limited in time and since the critical expertise tends to be strongly personalized. Experience is the only thing that really matters!

Wishing you smooth migrations!