Dataset · Hugging Face

commoncrawl/host-index-testing-v2

Name: host-index-testing-v2
Creator: commoncrawl
Keywords: task_categories:text-generation, size_categories:1B<n<10B, format:parquet, modality:tabular, modality:text, library:datasets, library:dask, library:polars, library:mlcroissant, region:us

Common Crawl Host Index v2 GitHub: https://github.com/commoncrawl/cc-host-index Each crawl, we generate a Host Index, which aggregates information about each web hosted visited during the crawl. The information is aggregated from the Common Crawl columnar index, web graph, and raw crawler logs. Quickstart The dataset is Hive-partitioned on crawl (data/crawl=CC-MAIN-2025-18/*.parquet). Open the whole dataset once, then filter with WHERE crawl = '...': because… See the full description on the dataset page: https://huggingface.co/datasets/commoncrawl/host-index-testing-v2.

View on Hugging FaceReport an issue

Author / organization	commoncrawl
Downloads	6K (as reported by Hugging Face)
Likes	1
Created	3 November 2025
Last updated at source	15 September 2026

Embed this entry

<iframe src="https://silytics.si/embed/item/ffaace2d-45f0-4315-8d8f-6e30e01d8604" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

More from commoncrawl

commoncrawl/gneissweb-annotation-url-testing-v1Dataset
commoncrawl/statisticsDataset
commoncrawl/commonlid-resultsDataset
commoncrawl/gneissweb-annotation-host-testing-v1Dataset
commoncrawl/llms.txtDataset
commoncrawl/web-graph-embeddingsDataset

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 16 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.