Data Quality

Data quality describes how far a dataset can be trusted for the decision it is meant to support, usually assessed across accuracy, completeness, consistency, timeliness and validity. In web data specifically, quality degrades from the outside rather than from within. A retailer changes a page layout, moves a price into a script block, adds a regional variant or renames a field, and a collection process that was correct yesterday starts returning partial or wrong values today without failing outright. That silent mode of failure is what makes continuous validation necessary. A pipeline that runs successfully and delivers a file on schedule tells you nothing about whether the numbers inside it are still right.

Why it matters

• Determines whether a dataset gets used, since a team that has been burned once will revert to manual checks and keep doing them long after the fault is fixed
• Fails quietly in web data, where a broken selector returns an empty field rather than an error, so absence of alerts is not evidence of health
• Compounds downstream, because a bad price feeding a repricing rule or a MAP alert produces a wrong action rather than a wrong number

How it is used

• Field-level validation against expected type, range and format on every run
• Coverage and fill rate checks comparing records returned against records expected
• Anomaly detection on values that parse correctly but fall outside plausible bounds

Data quality and pipeline reliability get treated as one thing and measure different failures. Reliability asks whether the job ran and the file arrived. Quality asks whether what arrived is correct. A pipeline can hit its uptime target every month while quietly delivering a field that stopped populating three weeks ago, which is why uptime alone is a weak proxy for either.