The Hidden Pitfall of Data Engineering

Data engineering is a critical component of any technology system, but it can be a hidden pitfall if not done correctly. Teams that hit this pitfall tend to misdiagnose it, blaming the model or the retrieval layer instead of the data engineering layer.

The Problem of Bad Data

Bad data can lead to confidently wrong answers, even if the model is correct. This is because the model is only as good as the data it is trained on. If the data is outdated, incomplete, or incorrect, the model will produce incorrect results.

A common failure mode in data engineering is the failure to validate data against the standards required by its consumers. This can lead to data that is not trustworthy, and can cause issues downstream.

Dimensions of Data Trustworthiness

There are four dimensions of data trustworthiness: correctness, freshness, consistency, and lineage. Correctness refers to whether each record conforms to the shape and rules it is supposed to. Freshness refers to whether the data is still current relative to its source. Consistency refers to whether the same fact reads the same way everywhere it is stored or indexed. Lineage refers to whether you can trace any output back to its source and every transform it passed through.

Tools like Great Expectations and Soda can help with correctness, by providing automated row and column-level validation. Freshness can be tracked by monitoring the time since the last successful update per source. Consistency can be checked by periodically cross-checking between downstream destinations. Lineage can be achieved by building a system that maps dependencies across different systems and datasets.

Real-World Examples

Companies like Uber and Netflix have built dedicated data quality and observability platforms to address these issues. Uber's Unified Data Quality platform supports over 2,000 critical datasets and detects around 90% of data quality incidents before they reach downstream consumers. Netflix's company-wide data lineage system allows anyone to answer where a dataset came from and what touched it along the way.

These companies have shown that it is possible to build systems that prioritize data trustworthiness, and that it is essential to do so in order to prevent issues downstream.

What to Do on Monday Morning

If you are running data engineering systems in production, the diagnostic question is not which model to try next or which retrieval architecture to migrate to. It is four narrower questions: Is the underlying data validated against the standards required by its consumers? What is the oldest piece of content currently being served with high confidence? Would two chunks of the same source ever disagree with each other in the same retrieval result? Could you trace where it came from if it turned out to be wrong?

If you cannot answer these questions, then the gap lies in the pipeline between your source systems and whatever your system reads from. That is a data engineering fix, not a model swap or a vendor migration.

Conclusion

Data engineering is a critical component of any technology system, and it is essential to prioritize data trustworthiness in order to prevent issues downstream. By focusing on correctness, freshness, consistency, and lineage, teams can build systems that produce trustworthy data and prevent confidently wrong answers.