Data Speaks for Itself: Data Validation – Data Accuracy Imposter or Assistant?

• Data validation and accuracy are distinct concepts, with validation focusing on detecting incorrect values and accuracy, requiring authoritative sources to verify correctness.
• Validation processes can miss inaccuracies, making methods like sampling and expert review crucial for identifying errors in records that pass validation.
• Continuous improvement of data quality management (DQM) relies on combining validation with broader techniques like Redman’s expert review process to close the gap between validation and factual accuracy.

Data validation and accuracy are often conflated in data quality management but serve different purposes. Validation identifies incorrect values through rule-based tests, while accuracy requires comparing data to authoritative sources. Validation alone cannot guarantee accuracy, as it may overlook errors in seemingly valid records.

Achieving true accuracy demands supplementary strategies beyond validation. Dr. Tom Redman’s “Friday Afternoon Measurement” involves experts reviewing records samples, even those that pass validation, to identify inaccuracies. This method highlights errors missed by validation and suggests new rules to enhance the process.

Organizations aiming to improve accuracy must adopt a holistic approach. This includes sampling, expert analysis, and iterative refinement of validation processes. The goal is to address validation limitations while implementing data profiling and quality assurance practices. Integrating advanced tools, such as generative AI, into expert review processes may further refine DQM efforts as data quality evolves.

Read the full article