The Discovery Problem
As data volumes grow, analysts struggle to find the right dataset for their analysis. Without a catalogue, they resort to emailing colleagues, duplicating datasets, and working with outdated versions.
What a Data Catalogue Contains
- Business Metadata: Dataset descriptions, ownership, and business use cases in plain language.
- Technical Metadata: Schema, data types, partition keys, and storage location.
- Operational Metadata: Last updated time, row count, data quality score, and freshness SLA.
- Lineage: Where did this data come from, and where does it flow downstream?
Popular Tools
Apache Atlas, DataHub (LinkedIn), Amundsen (Lyft), Alation, and Collibra are leading catalogue solutions across open-source and enterprise categories.