codeparrot/codeparrot-valid-near-deduplication
CodeParrot 🦜 Dataset after near deduplication (validation) Dataset Description A dataset of Python files from Github. We performed near deduplication of this dataset split codeparrot-clean-train from codeparrot-clean. Exact deduplication can miss a fair amount of nearly identical files. We used MinHash with a Jaccard threshold (default=0.85) to create duplicate clusters. Then these clusters are reduced to unique files based on the exact Jaccard similarity. Fore… See the full description on the dataset page: https://huggingface.co/datasets/codeparrot/codeparrot-valid-near-deduplication.
| Author / organization | codeparrot |
|---|---|
| Downloads | 914 (as reported by Hugging Face) |
| Likes | 1 |
| Created | 16 June 2022 |
| Last updated at source | 21 June 2022 |
Popularity, last 3 days
914 now (+91 since 2026-10-16). Counts as reported by the source, snapshotted daily by Silytics.
Recent changes
- 16 Oct 2026 Description: CodeParrot 🦜 Dataset after near deduplication (validation) Dataset Description A dataset of Python files from Github. We performed near deduplication of this dataset split codeparrot-clean-train from codeparrot-clean. Exact deduplication c → CodeParrot 🦜 Dataset after near deduplication (validation) Dataset Description A dataset of Python files from Github. We performed near deduplication of this dataset split codeparrot-clean-train from codeparrot-clean. Exact deduplication c
Changes detected by Silytics between refreshes of the source.
More from codeparrot
- codeparrot/codeparrot-cleanDataset
- codeparrot/github-code-cleanDataset
- codeparrot/github-codeDataset
- codeparrot/appsDataset
- codeparrot/codeparrot-clean-validDataset
- codeparrot/codeparrot-clean-trainDataset
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 18 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.