Skip to content

Dataset Card

Required data dataset_card
Agent Prompt Snippet
Include a dataset card that describes data sources, collection methodology, preprocessing steps, known biases, and licensing terms for every training dataset.

Purpose

A dataset card describes the data sources, collection methodology, preprocessing steps, known biases, and licensing terms so that downstream consumers can assess fitness for use.

This is a Required document — every project of this type should have one. Without it, the team risks misalignment, rework, or undetected issues that compound over time.

What Makes It Good vs Bad

A strong version of this document:

  • Defines clear data ownership, lineage, and quality expectations
  • Includes schema documentation with field-level descriptions
  • Specifies retention policies, archival rules, and deletion procedures
  • Documents data access patterns and query performance expectations
  • Addresses privacy requirements (PII handling, anonymization, consent)

Warning signs of a weak version:

  • Schema exists but fields are undocumented or ambiguously named
  • No retention policy — data grows indefinitely without governance
  • Missing data lineage — unclear where data originates and how it transforms
  • No privacy analysis for personally identifiable information
  • Query patterns undocumented, leading to performance surprises

Common Mistakes

  • Treating ‘we’ll figure out the schema later’ as a viable strategy
  • Not planning for data migration when schemas evolve
  • Ignoring data quality until downstream consumers report problems
  • Assuming all data access patterns are known at design time

How to Use This Document

Document schemas as living artifacts that evolve with the system. Include field-level descriptions, valid value ranges, and nullability constraints. Define a data classification scheme (public, internal, confidential, restricted) and label every data store accordingly. Plan for schema evolution from day one.

For AI agents: When modifying data models or queries, reference the data documentation to understand field semantics, access patterns, and privacy constraints. Ensure migrations preserve data integrity and backward compatibility.

  • Designing Data-Intensive Applications by Martin Kleppmann — Comprehensive guide to data modeling, storage engines, and distributed data systems.
  • The Data Warehouse Toolkit by Ralph Kimball & Margy Ross — The standard reference for dimensional modeling and data warehouse design.
  • Data Management at Scale by Piethein Strengholt — Modern approaches to data architecture, governance, and organizational data management.

Appears In