Dataset · Hugging Face

MC7ever/web-scraper-dataset

Name: web-scraper-dataset
Creator: MC7ever
License: other
Keywords: task_categories:text-generation, language:en, license:other, size_categories:10K<n<100K, format:parquet, modality:text, library:datasets, library:dask, library:polars, library:mlcroissant, region:us, web-crawl, multimodal, common-crawl

Web Scraper Dataset Whole-site crawl of 650+ domains across ~70 categories (news, wikis, fanfiction, education, NSFW, social, gaming, code forges, papers, shadow libraries and more). Common Crawl first (with retry/backoff), live crawlers (wget2/Playwright/Selenium/ requests) as fallback, uncapped per-domain page counts. Files MASTER_text.parquet — crawled page text (id, modality, language, source_domain, category, title, url, text, text_length, crawl_date)… See the full description on the dataset page: https://huggingface.co/datasets/MC7ever/web-scraper-dataset.

View on Hugging FaceReport an issue

Author / organization	MC7ever
License (as declared)	other
Downloads	138 (as reported by Hugging Face)
Likes	0
Created	30 August 2026
Last updated at source	29 September 2026

web-crawlmultimodalcommon-crawl

Embed this entry

<iframe src="https://silytics.si/embed/item/21e07596-4aba-4d16-8726-c4ce46d7d93a" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

More from MC7ever

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 20 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.