Metadata is information that describes other data. A scraped record’s metadata might include its source URL, collection time, language, response status, or parser version. These details help people understand where a value came from and whether it is suitable for a particular use.
Keep metadata useful
Choose fields that answer real questions: when was this collected, from which source, under what locale, and how was it transformed? Give each field a stable definition and avoid mixing metadata with source content. A timestamp generated by the crawler should not be confused with a publication date shown on the page.
Metadata supports filtering, debugging, and reproducibility, but more is not always better. Request headers, cookies, and account identifiers can contain sensitive information. Retain only what the task needs and define who can access it and how long it is kept.
If the schema changes, document the new field or meaning and preserve version information where older records remain. This prevents downstream users from interpreting a field inconsistently across runs.
Leave a Reply