Dataset · Hugging Face

datablations/oscar-filter

Name: oscar-filter
Creator: datablations
Keywords: size_categories:100M<n<1B, format:parquet, modality:tabular, modality:text, library:datasets, library:dask, library:mlcroissant, library:polars, region:us

this is the one where we build the suffix array for 25% Oscar and only deduplicate that part - by deduplication I mean removing any document which has an at least 100-char span overlapping with another document in the 25% chunk. This is very strict and preserves only about 20 million documents, so less then 5% of the full Oscar.

View on Hugging FaceReport an issue

Author / organization	datablations
Downloads	2.1K (as reported by Hugging Face)
Likes	0
Created	1 February 2023
Last updated at source	10 May 2023

Embed this entry

<iframe src="https://silytics.si/embed/item/72f821b5-af3a-4116-adb5-012d0116eccd" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Free, no tracking. More about feeds and embeds.

Popularity, last 2 days

2,149 now (+230 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.

More from datablations

datablations/oscar-dedup-expandedDataset
datablations/c4-subsetsDataset
datablations/python-megatronDataset
datablations/c4-filter-megatronDataset
datablations/scriptsDataset
datablations/c4-filterDataset

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.