Dataset · Hugging Face

crawlcheck/ai-consent-sample

Name: ai-consent-sample
Creator: crawlcheck
License: cc-by-4.0
Keywords: language:en, license:cc-by-4.0, size_categories:n<1K, format:json, modality:tabular, modality:text, library:datasets, library:pandas, library:polars, library:mlcroissant, region:us, ai-consent, robots-txt, content-signals, llms-txt

CrawlCheck: AI consent data (sample edition) What each site's robots.txt tells named AI agents, its Content-Signal line and llms.txt, signed; the free sample edition. Mirrored from crawlcheck.io/datasets/ai-consent by the CrawlCheck Worker; the site is the canonical copy and is updated first. Measured by CrawlCheck's own observer. Free under CC BY 4.0: credit "CrawlCheck" with the URL of the file. Catalogue of every dataset: https://crawlcheck.io/datasets How the data is made:… See the full description on the dataset page: https://huggingface.co/datasets/crawlcheck/ai-consent-sample.

View on Hugging FaceReport an issue

Author / organization	crawlcheck
License (as declared)	cc-by-4.0
Downloads	0 (as reported by Hugging Face)
Likes	0
Created	21 October 2026
Last updated at source	21 October 2026

ai-consentrobots-txtcontent-signalsllms-txtcrawlcheck

Embed this entry

<iframe src="https://silytics.si/embed/item/c83cde45-2b14-45e4-8199-fdb7a24fe85d" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

More from crawlcheck

crawlcheck/ai-crawlersDataset
crawlcheck/resolve-snapshotDataset
crawlcheck/agent-safety-benchmarkDataset
crawlcheck/vendor-benchmarkDataset
crawlcheck/census-weeklyDataset
crawlcheck/rights-feedDataset

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 21 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.