PREPARING FOR YOUR ARCHIVE MIGRATION. A Best Practices Guide TECHNICAL PAPER

Transcription

1 PREPARING FOR YOUR ARCHIVE MIGRATION A Best Practices Guide TECHNICAL PAPER

2 Executive Summary archiving solutions are a mainstay of the enterprise environment, particularly those that are in regulated industries or subject to ongoing ediscovery requirements. These environments have been subject to the same changes that have rippled through other facets of the online business world. In particular, the maturation of cloud-based archival and messaging platforms has seen organisations move to take advantage of their enhanced capabilities. archive migrations are complex undertakings and there are numerous risks associated with poorly planned migrations. For instance, manual migrations are often hampered by the native tools of legacy archive platforms with out-dated APIs, making extraction slow and difficult. As such, Insentra have prepared this paper to illustrate in greater detail the challenges that may arise in such migrations. Some common causes of data migration failure that can be avoided with a little forward planning include: The appearance of unanticipated or unknown data formats Insufficient or absent documentation for legacy systems Corruption of legacy data Breaking of chain-of-custody Legacy data that does not mesh well with the new platform Dramatic underestimation of migration time and cost In this overview, we offer key recommendations on preparing for the task, what risks to be aware of and how best to measure the success of your migration project. We trust this information is of value for your organisation as it embraces the benefits of a well-orchestrated transition to your target platform.

3 Simply put, an archive migration is the process of copying the content of an existing archive platform to a new target platform. Reasons you may need to Perform an Migration Despite sounding straightforward in concept, archive migrations can be an involved and lengthy process. Each organisation will have different requirements that will define the parameters and outcomes of the task and it is these nuances in which the devil lies. Many factors create the need for migrations, including: Changing the underlying storage platform of the current archive. Perhaps the current platform has reached end-of-life (EOL) and is no longer supported. It may have recurring issues that are unable to be resolved or its feature-set no longer meets your organisation s needs. Consolidation of environments as in the event of a merger or acquisition. When organisations merge, often there is an archive solution that needs to be consolidated. Maintaining multiple instances of the same archiving platform, or a few different archiving platforms, results in high management costs and an inefficient use of resources - not to mention a poor user experience. Splitting a subset of the content to another location (de-merger) OR Migrating to the cloud. Software as a Service (SaaS) archive solutions (i.e. Google, Mimecast, Office365, Symantec EV.Cloud etc.) present organisations with the option of no longer deploying and managing their own archive solutions. This can result in cost-efficiencies given the benefits of unlimited storage, often at a fixed per-user price. The proprietary nature of legacy archive platforms means that moving between different platforms is not as simple as just copying files. archive migrations can be completed either manually, or can be automated via the use of specialist third-party software tools. Migration Ingestion Process to O365 Cut over to mailboxes Select group of mailboxes to migrate Migrate Data Delete Shortcut Ingest to O365 Confirmation of ingestion Provision online mailbox We will be looking at the value of automated solutions shortly, but for now, let s tackle the essential aspects you ll need to consider before you delve too deep into your data. 3

4 What should I Plan for When Migrating Ensuring a smooth migration relies on your organisation getting the details right. The following factors will have a direct impact upon the success of the process: 1. Know where you re coming from, and where you re going It s crucial to have an in-depth understanding of your legacy archive environment. How far does the archive go back? What types of data did it capture (i.e. s, calendar appointments, attachments etc.)? What types of metadata did it capture? How much of this data do you need to transfer? This information will provide insight into what data you will be migrating and whether your archive infrastructure will be able to accommodate the extraction process. Understand also your target environment. Will it be cloud, or on-premises? What types of data formats will it accept and what is its maximum rate of ingestion? 2. Assessing the particulars Conducting an adequate pre-project assessment isn t just a good idea, it s a must if you want to effect a successful migration. It should provide you with a detailed picture of how much data exists in your archive, whether there is data - junk, perhaps - that can be discarded rather than migrated, and whether there are PSTs to include. Crucially, a pre-project assessment will provide insight into any legal and compliance risks that may present including whether chain-of- custody needs to be maintained (see box-out). Other questions it should resolve is the overall approach i.e, manual or automated migration, project costs and timeframes. 3. Choosing your migration pathway There are a number of different options when considering an archive migration, each with their own strengths and weaknesses. Parallel Systems and No Migration: A parallel process that keeps both old and new systems running concurrently allowing the old archived data to be expired over time and not migrated. Downsides: End-users have two systems to search for content, existing shortcuts will not always work, and ongoing support and maintenance costs with maintaining the legacy system. Benefits: the new archive is available as soon as possible at minimum expense and the legacy archive is accessible and can be quickly reverted to in the event of problems. 4

5 Selective Migration: Also known as a partial migration. A smaller group or subset of data is migrated rather than the entire archive. Benefits: The ability to minimise the volume of data being migrated, reducing costs and enabling end users to access their data from a single location. Downsides: To access ALL the data, two locations will still need to be referenced. Full Migration: All legacy data is migrated using a phased approach and the old system is decommissioned following verification of the migration. Benefits: Old platform can be decommissioned and all data is located in a single place. Downsides: Change for the organisation, upfront costs. 4. Plan the work, work the plan Once there is a basic understanding of the task ahead, it s time to formulate an overview of what should happen, when, and who is responsible for its implementation. It should also feature a realistic breakdown of the estimated resources needed and a clear statement of purpose around why the migration is being undertaken. Solicit advice from your compliance department and integrate their recommendations into your plan. Additionally, ensure that you communicate that plan to all parties likely to be affected to ensure everyone has a clear picture of what to expect, when to expect it and who to contact should the unexpected occur. 5. Cover your bases with a recovery plan Finally, have a robust contigency plan, should a problem emerge that cannot be readily addressed. Not having an option to roll-back to a reliable known state could prove costly and incredibly disruptive for your end-users. WHAT IS CHAIN-OF-CUSTODY? This refers to the reliable recording of processes and procedures that occur while evidence - physical or electronic - is being captured, held, transferred, or disposed of. For organisations with strict industry compliance regulations and internal data management policies, maintaining chain of custody is vital. When migrating, organisations must be able to demonstrate that the data has not been altered in anyway as it transits from one archive platform to another. Automated migration tools and services include a complete auditing of the migration process, which can demonstrate complete Chain of Custody for data that is subject to legal or regulatory requirements. 5

6 Common Questions About Archive Migrations Should I opt for a manual or automated migration? When assessing the best strategy for migrating your archives, one central question that needs to be addressed is whether to take a manual or automated approach. Both manual and automated approaches to migration share a common goal: the transferral of data from one platform to another. Manual migrations however have some very real limitations and inherent difficulties. In a manual migration, all legacy data is extracted to disk as the first step and then subsequently imported to the target platform. This approach requires the provision of the necessary disk space for the extraction which can be anywhere up to four times the size of the source archive. Additionally, manual migrations are prone to human error, incredibly slow and cumbersome, compromise chain of custody, present risks to data integrity, and require close monitoring. Indeed, many organisations find it so troublesome that the project stalls and is often never completed. Automated migrations leverage specialist migration middleware to simplify and streamline the process into a one-step data migration, directly from the source to the target platforms. This approach takes out the human element, is significantly faster and provides for chain of custody, ensuring a defensible migration. There are some use cases where a manual migration is appropriate. However, when chain of custody is critical and/or where more than 500 archives are required to be migrated, an automated migration is almost always required. But what about PST files? Can t I just use them to migrate to a new platform? PSTs have proven a valuable tool in times past and, on the surface, offer a good interim format for transferral of data. However, typically the volumes of data and numbers of archives render extracting to PST first as generally a not fit-for-purpose approach. In the majority of cases, manual migrations using PST files are slow, and present a higher risk of data loss through user error. Moreover, they do not offer any error logging or auditing capabilities and are unsuitable for migrations featuring a large number of mailboxes. It may be that your target platform can only receive data in PST, EML or MSG format. Don t fear in such instances, an automated migration will still enhance the speed of the extraction, while providing an audit log of all messages for compliance purposes. Automated migrations leverage software tools that can provide detailed reports to show 1:1 mapping of the ID of the item in the source archive and the ID of the new item as it is moved to the destination archive, thereby granting the ability to show a complete chain-ofcustody for the data. Even in cases where data must be changed for it to be ingested into a new platform, all changes are logged and identified, proving that the data itself has not been altered. This audit trail can be seen as defensible and means that old archive platforms can be decommissioned without legal concerns. Most importantly, automated migration tools are the only solution available that allows the preservation of BCC recipients and distribution list information from journal mailbox archives. These messages may be part of legal proceedings and/or may be subject to regulatory requirements, and their loss could become costly and potentially damaging to brand and reputation. 6

7 There are many issues with a manual PST migration, including (but not limited to): Slow and manually intensive. Typically, native archive extraction and import tools are single-threaded and need to be manually overseen. Seldom can multiple mailboxes be scheduled to run and they certainly cannot be left unattended. The impacts upon resources such as time, staffing and budget are significant. Little to no error management. If a PST extract or import fails, the process generally stops with no indication of the underlying problem. Rarely is there the ability to either identify and skip the items that fail, or to resume where the process left off. Little to no logging or auditing. When extracting to PST there is seldom a record of which data has been moved. All checks must be done manually to ensure that all messages in the source have been extracted to the target which is extremely time-consuming and prone to human error. PSTs risk the security and integrity of your data. Coupled with the lack of built-in checks, the multi-step process involved in PST migrations means that the extracted data needs to be held on interim storage during which it is open to tampering or corruption. Chain-of-custody is difficult to maintain, making a PST-driven migration unsuitable for organisations that have compliance or regulatory requirements. Loss of compliance data. When using a journal archive it is likely that envelopes containing BCC and distribution list recipients will have been stored in the archive and will require recreation following extraction. Manual extraction to PST files does not allow for the recreation of this envelope information. Accordingly, BCC and distribution list data is lost and cannot be searched in the target archive. Large PST files are prone to corruption. The standard tools shipped with an archive product often do not allow for the split of a mailbox into multiple small-sized PST files. As such, when extracting large mailbox or journal archives, PST files can become quite large and prone to corruption. Where this occurs, the source data will need to be extracted again and often the issue will recur. Requires interim storage. Most archive platforms provide single instance storage, i.e. only one copy of a message is stored in the database regardless of the number of recipients. For example, if a mail message with a 10MB attachment was sent to 15 people, it will have been stored in the archive as one 10MB message. Depending on the source archive and/or storage platform, this message may also have been compressed. When extracting this data to PST, a copy of the message will need to be extracted into each recipient s PST file. Therefore, the message that was previously 10MB will now become 150MB when extracted to 15 PST files. This loss of single instance storage often means that the PST extraction is, at minimum, 2.5 times the size of the original legacy archive. Limited or no flexibility in what is extracted. When extracting to PST files, typically all mailbox data will be extracted without an ability to filter or be selective over what is extracted and then ingested to the target. 7

8 Why do journal archives require special care? The majority of organisations capturing into Exchange journal mailboxes tend to do so for compliance reasons and typically use the Envelope Journaling feature. This feature was developed by Microsoft as a way to preserve vital header information including the BCC recipients and the expanded members of any distribution lists. As mentioned in the previous section, preservation of this data and making it available for access when performing e-discovery is pivotal to providing a compliant and defensible archive migration. Importantly, different archive platforms store the envelope information in different ways. Some may store the information separately in the archive index while others may store it in the archive store itself. This creates a requirement to recreate the envelope information at the time of migration - something that may not be (and often is not) supported by the native extraction tools of the source archive. Additionally, journal archives tend to be very large data repositories after all, they contain every mail sent to and from people in an organisation. This makes manual extraction to PST files very slow. Moreover, with many archive platforms, it is not actually possible to manually extract the envelope information nor to extract each message for each user. Migration software tools allow journals (and large mailboxes) to be split into a number of separately handled virtual mailboxes of a user-defined size for better allocation of computing resources and also address the expansion of the envelopes. If PST must be the target platform (which is OK) an automated migration will ensure that the PST files are of a manageable size and contain all of the source objects including the BCC and distribution list information contained in the envelopes. How will shortcuts be handled? Will they work in the new environment? Many archive platforms provide the ability to leave a shortcut or stub in the mailbox typically a smaller footprint of the original mail object (perhaps the attachments are stripped). These items need to be addressed as their own type of message. Importantly, If shortcuts are migrated as a part of the primary mailbox migration, there is a very good chance they will NOT work or will stop working after a period of time. Shortcuts need to be handled with kid-gloves and either removed or converted, the latter of which can only be done via an automated approach. An automated migration service ensures that shortcuts can be seamlessly converted as part of the migration allowing them to work with the new archive environment. Alternatively, they can be removed if they are no longer needed. 8

9 How long will our migration take? This is one of the more difficult yet most common questions. Every environment is configured and performs differently with the main variables affecting migration performance being: Available network bandwidth Speed of the storage subsystem on which the legacy archive sits as well as the destination storage The ingestion performance of the target archive system The scheduling of other projects such as the commissioning of the target environment The organisation s ability to transform When performing automated migrations, it is typical to refer to speed in terms of messages per second. The average performance of an automated migration is approximately thirty messages per second. However, the speed at which a migration will take place is often not determined until the migration has commenced and core data has been gathered. How will I know if my migration is a success? The key indicators of a successful migration are as follows: More than 99.9% of the readable data has been migrated Chain-of-custody has been maintained Timeframe to migrate data was reasonably met given health of the environment Internal resources have not been overly consumed by the project Corrupted data has been eliminated via the migration Project performed cost effectively Minimal impact on mail and data retrieval and the daily operations of staff Archived data has been effectively migrated into the new platform and support is in place for future data management and retrieval It is important to note that a manual extraction cannot address the bulk of these success criteria - a significant reason why more organisations adopt an automated approach to archive migrations. Failed or Skipped Items When migrating archive data, more than likely there will be failed or skipped items. There are two failed data categories: temporary and permanent. Temporary failures are usually due to environmental issues such as poor network bandwidth or high load on the legacy archive. In such cases, automated migration tools will automatically re-process the relevant item(s) a specified number of times and/ or at a different time of day until the object has been migrated. Permanent failures are usually attributable to preexisting problems in the source archive (i.e. not caused by the migration process). As such, it is likely that these items would not have been readable by a user or any audit/ediscovery process. This kind of failure happens fairly infrequently - typically about % of the overall archive content. Skipped items tend to occur when there are items in the source platform that are of a type or format not supported by the target and where these items cannot be altered in order to be ingested. These issues typically occur when migrating between Lotus Notes and Microsoft Exchange messaging platforms. 9

10 Can I perform my own migration, or would I benefit from a migration partner? Theoretically it is possible to perform your own migration, however experience has shown that even automated migrations require the involvement of an experienced and dedicated archive migration specialist. Such specialists are certified in the software tool being leveraged to complete the migration and know the traps to be avoided. Thorough knowledge of both the source and the target is critical. Each migration is different with many configuration permutations and combinations that need to be tested to ensure the optimum migration outcomes and throughout. A migration services partner will have the necessary skills and experience to deliver a successful outcome. Often the migration partner will provide an end-to-end service, ensuring that all source data has been migrated to the target. Some organisations consider contracting a specialist firm to provide design, implementation and initial user migrations, then seek to perform the balance of the migration themselves. Whilst this is a viable approach, it is not recommended. During migration, issues will inevitably arise that require expertise in the source and target platforms as well as in the tools to identify and isolate any problems. Troubleshooting these problems without having been a part of the process often takes significantly longer than if the migration specialist provides an outcome based service. Often, the additional cost of having a partner perform the end-to-end migration versus doing a part yourself is marginal, with the majority of organisations preferring to leave the migration to specialists who can take the risk and own the project end to end (i.e. all readable data migrated successfully). 10

11 Insentra s Automated Migration Approach Our Automated Migration Service methodology has successfully assisted many organisations to reach their new archive destination faster. Our approach has been put through its paces and our extensive experience has cemented a process that delivers a transparent and accountable migration. We are focused on delivering the agreed business outcomes of your organisations, following our five phase methodology: Establish ENGAGE Gather CAPTURE PLAN RECONCILE Phase One - Establish In the Establish phase we consider the objectives and requirements of the project and develop a bespoke roadmap to achieve the agreed migration milestones. We prepare a detailed Migration Plan (the archive migration equivalent to a design), taking into consideration how the archive migration project will fit in with the overall archive/mail consolidation project. Validate TEST Phase Two - Gather Once the Migration Plan has been formally agreed, the Gather ohase commences. In this phase, the migration infrastructure is built and tested prior to facilitating a gather (audit) of the existing archive. This gather process captures all possible data that can be migrated from the source archive. Insentra performs a detailed reconciliation of the archive data against the index built throughout the gather process to ensure all data has been accurately captured. Note: This exercise is absolutely critical to ensure a successful migration of all source archive data and acts as a checkpoint to ensure that the migration toolset has accounted for all data. Migrate PILOT Assurance REVIEW MIGRATE HANDOVER 11

12 Phase Three - Validate In the Validate phase, we conduct a number of tests to ensure the higheset performance can be achieved during the migration. We conduct a sample extraction to PST/NSF and to the target platform (note that the data is copied and not moved, making the process non-intrusive and transparent to the end-user). This testing ensures the highest possible migration speeds whilst providing validation of both chain of custody and data integrity. It is at this stage that more accurate estimations can be made about the duration of the migration project. Phase Four - Migrate In the Migrate Phase we begin by conducting a pilot in the live migration environment to confirm that the approach taken in the Validate phase will support the entire migration. We then commence a live migration by dividing mailboxes into manageable groups to rapidly move through the process. Phase Five - Assurance In the Assurance phase, we perform a review of all the migrated data, ensuring chain of custody and data integrity have been maintained in accordance with the project s requirements. During handover we discuss the migration reports, message failures and infrastructure decommissioning. The five-phase process ensures that your data is always protected, and the approach has successfully assisted countless organisations reach their new archive destination faster. Our methodology is extensible and can be applied to both simple and the most complex requirements. Helping you Make the Right Call Archive migrations are complex processes and the approach taken must be a considered one. All requirements must be well understood and all implications deliberated in order to facilitate a successful outcome. As such, we recommend: Adequite pre-planning, oversight and resource allocation for the task In most cases an automated migration to ensure chain-of-custody and a cost-effective outcome, versus a manual migration that is slow, noncompiant, and prone to error Partnering with an experienced migration specialist who can offer guidance and support, removing the difficulties of the process and minimizing risk to the organisation. There is no need to get archive migration wrong and every reason to get it right. The risks to your data of a poorly conceived migration are simply not worth it. 12

13 Knowing your Next Steps Archives are a necessity helping us to manage the explosion of data that is typical of today s information-rich environment. When properly implemented, cloud-based archival storage can provide a great boon for efficiency. As such, it pays to get it right. Always seek the best advice and ensure it gets done correctly from the get-go. Know your reasons for migration. Know what data needs to be moved, and how best to move it. Identify any limitations in your infrastructure, and what your compliance requirements are. Knowing where you re coming from will make it that much easier to see where you are going with your migration. Any questions? Insentra can answer them. 13

14 ABOUT INSENTRA Together we are better. Insentra is one of a kind. We believe that collaboration and genuine partnerships create a powerful synergy. Founded in 2010, we developed a unique IT channel model of engagement, which enables the efficient delivery of IT solutions for our vendors, partners and their clients. We are the only organisation in Australia with significant experience in data migration. Over the last five years we have migrated over a petabyte worth of data for our customers. With staff all over the world, we are ready and able to walk you through a migration in a way that is efficient and cost effective for your business. t e info@insentra.com.au Level 3, 110 Pacific Highway St Leonards NSW 2065 Australia insentra.com.au 14