Loading datasets...
Release Date: 5/14/2026
Format: tsv
Size: 59.38 MB
CommonLID is a community-created language identification (LID) benchmark. CommonLID consists of web text manually annotated for the language that it is written in. CommonLID contains annotations for 109 languages, where 78 of those languages have at least 100 lines of data.
Restrictions/Special Constraints
CommonLID is intended for evaluation only, so please do not use it to train LID models or other AI models. Please do not re-host CommonLID in places where it could be picked up by web crawlers CommonLID is a research dataset, so if you use it in your research, we kindly ask you to cite our work using the citation information provided below in the Citation section.
Forbidden Usage
Cannot be used for training models
Ethical Review
CommonLID is intended as a domain-specific evaluation for LID models for web data curation.
Intended Use
Evaluating Language Identification Systems
CommonLID is a community-created language identification (LID) benchmark. CommonLID consists of web text manually annotated for the language that it is written in. CommonLID contains annotations for 109 languages, where 78 of those languages have at least 100 lines of data. The number of lines available for each language is provided in Appendix A of the preprint.
Method details are in our preprint: CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data (2026). CommonLID was created as part of a shared task at the Workshop on Multilingual Data Quality Signals (WMDQS) at COLM 2025. We invited members of the community to help annotate web data in their languages. Native speakers created line-level LID annotations for over 350,000 lines of web data. Annotations were validated by an expert NLP researcher, familiar with several different writing systems.
All contributors who annotated at least 100 documents (or all of the documents available in their language, if there were fewer than 100 documents available) were invited to be authors on the dataset and the paper preprint.
CommonLID proves to be a more challenging LID dataset than existing ones. Most models perform worse on CommonLID than other datasets. This suggests that current evaluation datasets may overestimate LID performance in the web domain.
CommonLID is composed of data sampled from the CC-MAIN-2024-22 and CC-MAIN-2025-05 crawls from Common Crawl, as well as MADLAD-400 which is a dataset derived from Common Crawl. As such CommonLID is released under the Common Crawl Terms of Use. CommonLID is intended for evaluation only, so please do not use it to train LID models or other AI models. Please do not re-host CommonLID in places where it could be picked up by web crawlers CommonLID is a research dataset, so if you use it in your research, we kindly ask you to cite our work using the citation information provided below in the Citation section.
CommonLID is intended as a domain-specific evaluation for LID models for web data curation.
CommonLID only includes data for a small subset of the world's languages and the amount of data available for each language is not the same for each class. Please see the preprint for our recommendations about how to conduct fair evaluation and cross-model comparisons.
The data in CommonLID is sourced from unfiltered web data and may contain offensive, harmful, or NSFW content.
@article{suarez2026commonlid,
title={CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data},
author={Ortiz Suarez, Pedro and Burchell, Laurie and Arnett, Catherine and Mosquera-G{\'o}mez, Rafael and Hincapie-Monsalve, Sara and Vaughan, Thom and Stewart, Damian and Ostendorff, Malte and Abdulmumin, Idris and Marivate, Vukosi and others},
journal={arXiv preprint arXiv:2601.18026},
year={2026},
url={https://arxiv.org/abs/2601.18026}
}
CommonLID was created in partnership with the Common Crawl Foundation, ML Commons, EleutherAI, and Johns Hopkins University.