Dataset · Hugging Face

open-index/ccrawl-recrawl-urls

Name: ccrawl-recrawl-urls
Creator: open-index
License: odc-by
Keywords: task_categories:text-generation, task_categories:text-retrieval, task_categories:other, license:odc-by, size_categories:1M<n<10M, format:parquet, modality:tabular, modality:text, library:datasets, library:dask, library:polars, library:mlcroissant, region:us, common-crawl, web-crawl

Common Crawl URL Recrawl Live refetches of pages from Common Crawl's URL index, with the body inline and the text already extracted What is it? Common Crawl publishes which URLs it saw and when, but the page bodies live in WARC archives that are awkward to query and are as old as the crawl that made them. This dataset takes the URL index for a single monthly crawl and fetches the pages again now, storing each response as one Parquet row with the body, the… See the full description on the dataset page: https://huggingface.co/datasets/open-index/ccrawl-recrawl-urls.

View on Hugging FaceReport an issue

Author / organization	open-index
License (as declared)	odc-by
Downloads	460 (as reported by Hugging Face)
Likes	0
Created	20 August 2026
Last updated at source	22 August 2026

common-crawlweb-crawlrecrawlhtmlmarkdowntextparquetopen-dataurl-index

Embed this entry

<iframe src="https://silytics.si/embed/item/631d58aa-9975-4d9d-92d7-b8e32cd9d4a9" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

Popularity, last 3 days

460 now (+16 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.

More from open-index

open-index/hacker-newsDataset
open-index/arcticDataset
open-index/open-githubDataset
open-index/open-libraryDataset
open-index/open-wikipedia-markdownDataset
open-index/ccrawl-recrawl-domainsDataset

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 18 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.