Dataset · Hugging Face

eduagarcia/CrawlPT_dedup

Name: CrawlPT_dedup
Creator: eduagarcia
Keywords: task_categories:text-generation, language:pt, size_categories:100M<n<1B, format:parquet, modality:tabular, modality:text, library:datasets, library:dask, library:mlcroissant, library:polars, arxiv:1911.02116, arxiv:2201.06642, arxiv:2107.06499, region:us

CrawlPT (deduplicated) CrawlPT is a generic Portuguese corpus extracted from various web pages. This version is deduplicated using MinHash algorithm and Locality Sensitive Hashing, following the approach of Lee et al. (2022). The raw version is also available here. Dataset Details Dataset is composed by three corpora: brWaC, C100-PT, OSCAR-2301. brWaC: a web corpus for Brazilian Portuguese from 120,000 different websites. C100-PT: Portuguese subset from CC-100.… See the full description on the dataset page: https://huggingface.co/datasets/eduagarcia/CrawlPT_dedup.

View on Hugging FaceReport an issue

Author / organization	eduagarcia
Downloads	2.5K (as reported by Hugging Face)
Likes	8
Created	2 February 2024
Last updated at source	14 March 2024

Embed this entry

<iframe src="https://silytics.si/embed/item/fa8af0bc-1ce7-4d0f-9b45-756917f1bd4d" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

Popularity, last 3 days

2,499 now (+52 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.

Connected in the knowledge graph

eduagarcia/RoBERTaLexPT-baseModel · used to train
eduagarcia/RoBERTaCrawlPT-baseModel · used to train

Explore this entry in the graph

Recent changes

16 Oct 2026 Description: CrawlPT (deduplicated) CrawlPT is a generic Portuguese corpus extracted from various web pages. This version is deduplicated using MinHash algorithm and Locality Sensitive Hashing, following the approach of Lee et al. (2022). The raw versio → CrawlPT (deduplicated) CrawlPT is a generic Portuguese corpus extracted from various web pages. This version is deduplicated using MinHash algorithm and Locality Sensitive Hashing, following the approach of Lee et al. (2022). The raw versio

Changes detected by Silytics between refreshes of the source.

More from eduagarcia

eduagarcia/tokenizer_benchmark_resultsDataset
eduagarcia/LegalPT_dedupDataset
eduagarcia/enem_challengeDataset
eduagarcia/portuguese_benchmarkDataset
eduagarcia/cc_news_pt_v2Dataset
eduagarcia/mc4-ptDataset

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 18 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.