crawlcheck/rights-feed
CrawlCheck: AI-use declarations, top 10,000 What the top 10,000 sites declare about AI use, one signed line per site, for crawler operators to load before fetching. Each edition is a signed manifest plus gzip NDJSON parts. Mirrored from crawlcheck.io/spec/rights-feed by the CrawlCheck Worker; the site is the canonical copy and is updated first. Measured by CrawlCheck's own observer. Free under CC BY 4.0: credit "CrawlCheck" with the URL of the file. Each edition is a signed… See the full description on the dataset page: https://huggingface.co/datasets/crawlcheck/rights-feed.
| Author / organization | crawlcheck |
|---|---|
| License (as declared) | cc-by-4.0 |
| Downloads | 0 (as reported by Hugging Face) |
| Likes | 0 |
| Created | 21 October 2026 |
| Last updated at source | 21 October 2026 |
Recent changes
- 21 Oct 2026 License: none → cc-by-4.0
- 21 Oct 2026 Description: none → CrawlCheck: AI-use declarations, top 10,000 What the top 10,000 sites declare about AI use, one signed line per site, for crawler operators to load before fetching. Each edition is a signed manifest plus gzip NDJSON parts. Mirrored from cra
- 21 Oct 2026 Topics added ai-rights, ai-training-opt-out, robots-txt, crawler-policy, crawlcheck
Changes detected by Silytics between refreshes of the source.
More from crawlcheck
- crawlcheck/ai-crawlersDataset
- crawlcheck/resolve-snapshotDataset
- crawlcheck/vendor-benchmarkDataset
- crawlcheck/census-weeklyDataset
- crawlcheck/ai-consent-sampleDataset
- crawlcheck/finding-countsDataset
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 21 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.