4.6 Agile Data Migration
File migration is process intensive due to a large number of small files. This adds to the migration cost. Bulk file transfer is not an option where each file needs to be re-positioned to a different path in the target file store.
Iterative release
We completed the entire migration across three iterations. Each iteration delivers the most recent data to the UAT to be tested against the latest application release. Each iteration migrates only the new and modified files since the previous one. This is needed since the application is in active use even during the UAT phase. The last iteration ensures the delta files are migrated and then moved to the Prod after one round of smoke testing.
On the cut-off date, we had as few as 25,000 files left to be migrated (less than 2%). These were files created new or modified on the “live application” during the UAT time period and were processed within a few hours.
4.7 Testing and validation
Validation and testing of the migration are the most critical tasks. Approximately 50% of the overall migration effort is spent on validating the scripts. The following lists areas where the testing effort comes into play for the most part:
Validating the SQL migration scripts
- Ensure that all mapped fields and tables are covered and the source and the target row numbers are consistent.
- Validate that the foreign key references are consistent and accurate in the destination database
- Where the number of rows does not tally due to data quality issue, either should be resolved with input from the business team or signed off as exceptions citing the rationale.
- Maintain records of rows and columns exempted from migration for any reason with the rationale and the number of rows being affected.
4.8 The cut-over plan
Figure 8 is a representative illustration of the cut-over plan we followed
Cut-over plan steps
- The swim lanes represent the three environments: On-prem, UAT, and the Prod (the dev excluded for brevity). The staging is a part of the UAT where original data is present with access limited to approved users.
- During the pre-UAT phase, scripts are developed and tested on the dev, with the help of a small subset of the original data (masked).
- Where the data volume is huge, and migration and re-engineering is accomplished over multiple iterations; each iteration appends data to the staging.
- The re-engineered data is tested using scripts. Additional validation of the re-engineered data is done by connecting it with the new application to validate integrated functioning.
- The final iteration (this is managed during a brief cut-over period) migrates and re-engineers the data modified or created on-premise during the UAT time period.
Leave a Reply