crawlcheck/census-weekly
CrawlCheck: Agentic-web census, weekly The top sites on a fixed frame: crawler policy, machine files, declared capabilities and whether AI crawlers get what a browser gets, sealed once a week. Mirrored from crawlcheck.io/data/census/weekly by the CrawlCheck Worker; the site is the canonical copy and is updated first. Measured by CrawlCheck's own observer. Free under CC BY 4.0: credit "CrawlCheck" with the URL of the file. Catalogue of every dataset:… See the full description on the dataset page: https://huggingface.co/datasets/crawlcheck/census-weekly.
| Author / organization | crawlcheck |
|---|---|
| License (as declared) | cc-by-4.0 |
| Downloads | 0 (as reported by Hugging Face) |
| Likes | 0 |
| Created | 21 October 2026 |
| Last updated at source | 21 October 2026 |
Recent changes
- 21 Oct 2026 License: none → cc-by-4.0
- 21 Oct 2026 Description: none → CrawlCheck: Agentic-web census, weekly The top sites on a fixed frame: crawler policy, machine files, declared capabilities and whether AI crawlers get what a browser gets, sealed once a week. Mirrored from crawlcheck.io/data/census/weekly
- 21 Oct 2026 Topics added web-census, ai-crawlers, agentic-web, crawlcheck
Changes detected by Silytics between refreshes of the source.
More from crawlcheck
- crawlcheck/ai-crawlersDataset
- crawlcheck/resolve-snapshotDataset
- crawlcheck/vendor-benchmarkDataset
- crawlcheck/rights-feedDataset
- crawlcheck/ai-consent-sampleDataset
- crawlcheck/finding-countsDataset
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 21 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.