pathwren/ai-crawler-index
AI Crawler Index 150 web crawlers and AI user agents from 74 operators — what each one is for, what blocking it costs you, and the IP ranges its operator publishes. Plus a compiled user-agent regex and the union of 1997 IPv4 and 1062 IPv6 prefixes from 15 operator-published range files. Home: https://www.pathwren.workers.dev/c/huggingface-datasets/ · CC0 · no signup, no key. What this is, plainly This is an independent, non-commercial automated project. It is run… See the full description on the dataset page: https://huggingface.co/datasets/pathwren/ai-crawler-index.
| Author / organization | pathwren |
|---|---|
| License (as declared) | cc0-1.0 |
| Downloads | 370 (as reported by Hugging Face) |
| Likes | 0 |
| Created | 1 September 2026 |
| Last updated at source | 14 September 2026 |
Popularity, last 2 days
370 now (-1 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.