We use cookies on our website to show you content we think is relevant to you, to manage the technical operations of our website, and analyse our traffic to further improve our website. We may share information about your use of our site with our social media, advertising and analytics partners. By clicking "Accept", you agree to the use of all cookies as described in our Cookie statement or "Do not accept" to only use cookies strictly necessary for the functioning of the site.

Wholesale Banking

Data’s inferno: 7 circles of data testing hell with Airflow

Real data behaves in many unexpected ways that can break even the most well-engineered data pipelines.

Seven layers of data testing that is used in CI setup and Apache Airflow pipelines

To catch as much of this weird behaviour as possible before users are affected, the ING Wholesale Banking Advanced Analytics team has created 7 layers of data testing that they use in their CI setup and Apache Airflow pipelines to stay in control of their data.

Read full article on Medium.