Migrating Vignette to SharePoint: Part II – Findings

AccessECM was recently engaged to develop a plan for migrating a Vignette Collaboration system to a SharePoint Online.  In the first part of this series I provided an overview of the overall effort.  In Part II we review our findings based on a pilot migration project.

Export – Transform – Load (ETL)

ETL is a process that gained popularity in the 1970s as the amount of data and the need to warehouse it grew quickly.  The process allows organization to aggregate data from heterogeneous or homogeneous systems into a single repository.  Since each of the processes can takes time, it is common to execute the three phases in parallel.  The process has been proven effective with large amounts of data, which makes it ideal for our project.    

Export

For Export phase, we began by testing the Vignette’ built-in export tool called “DE”.  The tool was run against Vignette containers of varied size and found that it performed well for smaller containers, generating exports of content and metadata.  However, we saw obvious problems with larger exports.  Specifically, results for jobs exceeding 60 to 90 minutes were erratic, sometimes never concluding.  An additional issue with DE exports is that they must be manually triggered by an operator through the Vignette user interface.  Since our intention is to create an automated migration engine, this option is not adequate.

The second option is to develop a custom export tool that leverages Vignette APIs to generate exports.  We built a prototype version of this tool and tested it under different configurations.  We were able to execute exports of both large and very large containers (20-100 GB) with speeds ranging from 3 to 30 GB/hr.

Some other observations and notes about exports:

1. Exports can be scheduled and run programmatic without user intervention
2. Exports can be run in parallel (using multithreading) or throttled to reduce impact to users
3. Occasionally exports become unresponsive. As a workaround, most of these jobs can be stopped and resumed to continue the export.
4. Outputs can be stored inside any accessible network device, but this slows down the export significantly.  When possible, local storage should be utilized to store exports.
5. As a rule, very large exports (>25 GB) should be avoided by splitting them into smaller chunks.

Transform

The underlying philosophies for both Vignette and SharePoint are similar in that they are designed to store and serve content. However, there a variety of differences in how this is accomplished. A good example of this is where a file can be stored. In Vignette files can reside in containers or folders of any kind, while in SharePoint files are generally stored in specific objects called Document Libraries. To overcome these types of differences a conversion methodology (or “transformation”) needs to be developed. The list below is summarizes the areas where conversions are necessary.

1. Container Structure (Heirarchy) – As mentioned earlier, SharePoint is more limited in where content can reside as opposed to Vignette. A conversion methodology that transforms ARC folders and containers into SharePoint sites, document libraries and folders must be implemented.
2. Security Mapping – There are similarities in the way security is applied within the two systems. However, Vignette encourages users to implement very granular security policies directly to individual files. Although this functionality is technically available within SharePoint, it is highly discouraged due to its impact on performance. It is also time consuming to apply individual permissions during the loading phase. Instead we recommend an approach where security policies are applied only down to the document library level.
3. Emails – Vignette offers users convenient email functionality ranging from the ability to store individual emails to something similar to a shared mailbox. These emails are stored in a proprietary format that is not compatible with SharePoint. In order to migrate emails from Vignette, a process must be implemented to convert the emails into a SharePoint compatible format.

Note: In some cases, a conversion is impractical due to lack of similar features in SharePoint or the overall cost. A good example of this is metadata. Both systems are designed to store and display metadata, but a fully automated conversion process for migration is not practical. These should be handled on a cases-by-case basis.

Import (Load)

Imports into SharePoint consist of three separate activities: (1) creation of the SharePoint hierarchy (sites, sub-sites, and document libraries), (2) application of security permissions and (3) upload of content and metadata.

1. Creating a SharePoint Hierarchy – As described in the Transform section (above), a conversion process is require to remap Vignette containers and folders into appropriate SharePoint objects. During this portion of the load process, we used PowerShell (PS) scripts to build SharePoint sites, sub-sites and document libraries.
2. Apply Security Permissions – Once a SharePoint hierarchy is created, a process to apply security permissions should be executed. Again, we used PS scripts to accomplish this task. We do not recommend reproducing Vignette security below the Document Library level.
3. Content Upload – Once the SharePoint structure and security has been applied, content from the export phase should be imported. In our pilot we used Microsoft’s Migration API (designed for SharePoint Online) for this purpose. There are many variables that impact the speed of these imports and likewise many opportunities to optimize it. In our experiences we detected speeds ranging from a low of 500 MB/hr to as high as 6 GB/hr.

In Part III, we will review our recommendations for executing a bulk migration.

To learn how we can reduce the cost and duration of your SharePoint migration project, reach out to our experts.

Steven Pogrebivsky

Steve Pogrebivsky is an expert in information and content management systems with over 20 years of experience. Steve holds a BS in Computer/Electrical Engineering and an MBA in Information Systems from Drexel University.