Migrating to the clouds:

The Context and The Challenges

The on-premise application being modernised is used by 400 to 500 users across the country to manage artefacts related to court proceedings and to address regulatory compliance. The data is sensitive to the extent that the design and development team can have no visibility of the actual data. The same applies to the document repository. The confidentiality requirements necessitate that the migration is executed within the limits of these constraints.

The migration payload

The migration payload is of two types: Data and Unstructured files. The data is re-engineered to suit the new database schema of the target application. The unstructured data, mainly of .msg .pdf, .mp4, .mp3 and .doc files present in the on-premise drive is re-located to Azure blobs.

Data quality issues

During the analysis phase, we identified a variety of data quality issues. Over time, many fields in SQL DB were re-purposed or modified to accommodate changing needs, compromising the data integrity. At the same time, many fields in the database were made redundant since quite a few form fields were left unused by the business users.

Out of the 2.4 million files that had to be moved, only 770,000 files could be accessed through SQL data. The rest were practically “missing” from an application perspective.

A portion of the file artefacts comprises outlook messages. The subject lines of these messages contain special characters from Indo-European languages. While these messages are saved to the hard disk, the filename defaults to the email subject, and the special characters are not properly encoded in the filename by the operating system. This results in a disconnect between the filename and the mapping in the SQL DB, creating what could be termed an “untraceable file”.

Connectivity issues to the legacy server

The legacy data centre offers a minuscule 0.75mbps bandwidth for the file migration exercise. Transferring 2+ million files using an ETL pipeline is nearly impossible due to the large volume of files and the frequent interruptions. Bulk file transfer is not an option since each file had to be re-positioned to a new location in the target file store.

Hot migration with near-zero downtime

Even during the migration, the on-prem application is in active use. So, the final leg of the migration exercise needs to be completed with minimum downtime to the application while switching to the new one.

To overcome these challenges, we engineered an 8-point framework that
also went through many refinements during the migration exercise.

Our 8 Point Migration Framework

1. Migration strategy

Our migration strategy covers the entire cloud migration life cycle, with scope, plans, dependencies, and risks. An elaborate migration strategy enables breaking into the minutest aspects of the migration and re-engineering process. Visualising the end-to-end process and documenting the method help us socialise the idea across the development and the migration team, communicating the risks and issues to the client and getting a consensus across the board for the best stakeholder support.

Phased Migration

The migration is attempted in two distinct phases:

Phase-I migrates all critical business data needed to launch the new application so that users switch over to the new application sooner for early validation and benefits realisation.

Phase-II builds on additional features to the cloud-native application. This phase involves another round of data migration and re-engineering. However, the iterative approach designed for the first phase is generic and can be replicated for any number of subsequent migrations. At the same time, when additional phases are anticipated, it is critical to retain a copy of re-engineered data in the staging area for future use. Refer section 4.4.

The conceptual approach

The high-level migration approach is illustrated in Figure 4. Conceptually, the on-premise DB is lifted to the cloud environment and further reengineered to adapt to the destination schema.

Two-step migration process

The data migration is accomplished in two steps: The first entails an “as-is” migration of the on-prem data to Azure. This is archived for future reference and compliance. In the second step, this data is refactored to the new schema. This activity, since it is performed on Azure, helps leverage the efficiencies of the cloud platform.

While the binary copy activity in ADF is a standard and reliable method to copy files, due to bandwidth limitations, we employed PowerShell scripts to transfer files to a staging area on Azure before using the ADF pipeline for further processing.

Inventory file for meta-data based migration[1]

The meta-data based file migration relies on an inventory mapping file. This file maps the path of the source files to their new blob paths and additional meta-data that needs to be retained. The inventory is prepared during the analysis phase and helps trace lost files, improving data availability. The same list acts as a filter to exclude migrated files in future iterations.

2. Data and file analysis

Detailed analysis of the source and destination data models is a prerequisite even for the simplest of migrations.

Data Models

Data models of the source and destination databases help the mapping
activity by providing the visualisations of the schema. The source data
model is prepared by reverse-engineering the source database while the
destination data model is made available as part of the new DB design. To
highlight, the data models:

• Help the SQL developers rapidly understand the source and destination metadata.
• Assist the SQL script engineering by being a source of reference for the different tables in the database.
• Are critical in documenting the migration process and communicating with stakeholders at all levels.

Creating the file inventory

Analysis of the source files is based on an “inventory” of files within the file
store and all its sub-folders. We use PowerShell scripts to generate the list
(inventory) from the source exported into .csv format. An Azure function
generates the file inventory from the destination (blob). These inventory files
compare the status of the files in the source and destination during each
migration iteration.

3. Developing the migration scripts

Scripts (PowerShell and SQL) are employed at different stages of migration from the on-premise as well as in the ADF.

SQL scripts

SQL scripts are employed to re-engineer the source data to the destination schema. These scripts eventually form parts of the ADF pipeline that migrates data from the origin to the destination data repositories.

PowerShell scripts

PowerShell scripts are “powerful” to carry out various tasks in migration. They offer fine-grain control during the migration process than what is possible with an ADF Pipeline. We employed these scripts:

• To generate the inventory from the source file store (root folder)
• To copy the files from the source file store to a staging area within the
  destination cloud network. See the rationale in section 4.1

4. Meta-data driven file migration (re-engineer once)

There are two golden rules that we observe throughout the entire migration process:

• Avoid migrating or re-engineering files that are migrated once to prevent recurring data processing costs.
• Ensure re-engineering accuracy by providing the migrated data for
  UAT testing in each iteration. An untouched copy of the engineered
  data/files is maintained that is later moved to production.

Selective migration of files is made possible by the inventory file acting as a
filter to exclude previously engineered data in subsequent iterations. The
meta-data based migration remained critical in taking this approach [2].

“Closing in on the cut-over, as few as 2% of the files remained to be migrated, that we completed within a few hours. These were files created during the UAT or ones modified subsequent to the previous migration. Taking the last mile during the weekend, we accomplished the migration, with zero effective downtime.

The iterative approach provided a robust set of data for the users to test the application against in each iteration.”

4.5 The Agile team, Scrum process and roles

The daily scrum meetup we established to assess the progress established the cadence for a thorough review and improved data quality.

“The data migration is orchestrated after detailed analysis and end-to-end planning.”

“Our success is the outcome of getting together people with the right skills, clear responsibilities and appropriate representation from the dev and business teams.”

Resolving data quality issues

The representation of the development team in these meetings is crucial. The daily scrum with the application administrator is highly influential in resolving the data quality issues. Through proper involvement, timely communication and discussion, the team nails down issues and get the system admin to fix them on time.

Data security through masking

Our strict security protocols ensure compliance with the privacy laws. The data being highly sensitive, only the critical and mandatory fields are included in the mapping table and are visible to the team. Additionally, individual fields are masked using the SQL Server masking option, with only foreign keys and mandatory master data visible to the developers.

Pages: 1 2 3 4