Originally published on LinkedIn May 28, 2018.

Introduction

Data Quality issues haunt most of the organizations. Contradictorily though, when data sharing is prevented by law, research on this becomes a challenge in itself and it becomes imperative to build a solution to replicate the data problem to facilitate analysis.

A solution to this challenge is to simulate the original dataset to create an experimental framework to perform analysis. This is the outcome of a research to improve data quality by identifying duplicate customer records using simulated data.

Figure 1: The Conceptual Framework

The Framework

The conceptual framework is presented in figure 1.

Data Simulation: Data simulation is performed using any test data generation engines.

Problem Injection: Simulation of problem records is performed within the R code itself employed further for statistical analysis.

De duplication: The crux of the de-duplication process is the “Word Distance Measuring” algorithms currently encapsulated in many R packages.

Supervised Classification: The classification is performed using a pre-trained dataset statistically sampled from the original population truly representing the poor quality data.

Toolset: The test data is simulated using php laravel framework and “faker” algorithms subsequently loaded to postgresql db. Data modifications and statistical classification can be performed in R.

Conclusion

The initial version of the framework is able to perform de-duplication with above 98% accuracy.

Download the full study from Research Gate …

http://www.researchgate.net/publication/325259869_A_Controlled_Experiment_to_improve_data_quality