Datasets | Mozilla Data Collective

SdCorpus is a large scale monolingual Sindhi text corpus collected from publicly available web sources for NLP and language model development.

Sindhi Corpus

License: CC-BY-NC-4.0

Locale: snd

Task: LM

Format: TXT

Size: 321.00 MB

Hindi monolingual corpus of about 10 million tokens, compiled from multiple authors and including literary works and personal articles.

Hindi 10 Million Text Corpus

License: CC-BY-ND-4.0

Locale: hin

Task: NLP

Format: DOCX

Size: 21.91 MB

A 485k-word Tatar folklore corpus from 20th-century field recordings, selected from 5 academic volumes to prioritize contemporary linguistic usage.

Tatar Folklore Text Corpus

License: CC0-1.0

Locale: tt

Task: NLP

Format: TXT

Size: 1.40 MB

CLEAR Global

TWB Parallel Sentence kits - Kanuri (5k)

5,000 English–Kanuri parallel sentences from CLEAR Global's Gamayun mini-kit, translated by professional translators from Tatoeba source sentences.

License: CC-BY-4.0

Locale: kau

Task: MT

Format: TSV

Size: 358.46 KB

Longitudinal Kazakh news corpus from Radio Azattyq (2003-2026) with over 139,000 articles and 35M tokens.

RFE/RL Kazakh News Text Corpus

License: CC-BY-NC-SA-4.0

Locale: kk

Task: NLP

Format: TXT

Size: 126.81 MB

Anjuman e Katib

Anjuman-e-Katib Farsi/Persian Literature Corpus

This corpus is a collection of more than one million tokens of Farsi/Persian language. The corpus contains work of literature including novels, fictional and non-fictional, poems and many more. the data is shared with the approval of the authors and aims to support linguistic research, language technology development and its preservation.

License: CC-BY-NC-4.0

Locale: fas

Task: NLP

Format: TXT

Size: 2.82 MB

TAUSCompensated

🚧 Colloquial Parallel Text (English - Hausa) 🚧

49k native-reviewed colloquial sentences translated from US English into Hausa for fine-tuning conversational MT and AI models.

License: TAUS DUA

Locale: en-US, ha-NG

Task: MT

Format: TSV

Size: 2.93 MB

Longitudinal Chechen news corpus from Radio Marsho (2006-2026) with over 30,000 articles and 8.4M tokens.

RFE/RL Chechen News Text Corpus

License: CC-BY-NC-SA-4.0

Locale: ce

Task: NLP

Format: TXT

Size: 28.29 MB

A curated Marathi text corpus containing diverse written content for natural language processing, linguistic research, and language modeling tasks.

Marathi Blog & Literature Corpus

License: CC-BY-NC-4.0

Locale: mar

Task: NLP

Format: TXT

Size: 4.20 MB

Longitudinal Georgian news corpus from Radio Tavisupleba (2001-2026) with over 238,000 articles and 38M tokens.

RFE/RL Georgian News Text Corpus

License: CC-BY-NC-SA-4.0

Locale: ka

Task: NLP

Format: TXT

Size: 257.53 MB

A collection of annotated images capturing public space activities, traditional handicrafts, historical and cultural heritage exhibits in Java.

IND IKRAR - Craft and Cultural Heritage Multimodal Corpus

License: CC-BY-NC-SA-4.0

Locale: ind

Task: CV

Format: JPEG, WEBM, JSON

Size: 286.27 MB

A corpus of traditional Kazakh proverbs from the Babalar sözi series (vols 65-69), designed for NLP and LLM cultural competence evaluations.

Kazakh Proverbs Text Corpus

License: CC0-1.0

Locale: kk

Task: NLP

Format: TXT

Size: 592.14 KB

MDC DatasetsCompensated

🚧 Khowar Speech Translation Corpus 🚧

A Khowar-English speech translation dataset with approximately 9.5 hours of Khowar audio with English text translations.

License: MDC DLA-1.0

Locale: khw, en

Task: MT

Format: MP3, TSV

Size: 244.96 MB

Longitudinal Azerbaijani & Russian news corpus from Radio Azadlıq (2005-2026) with over 239,000 articles and 37M tokens.

RFE/RL Azerbaijani News Text Corpus

License: CC-BY-NC-SA-4.0

Locale: az,ru

Task: NLP

Format: TXT

Size: 211.65 MB

A 427k-word Kyrgyz folklore corpus of tales, proverbs, and aphorisms, digitized from 5 Bishkek academic volumes (2016-2017) for NLP tasks.

Kyrgyz Folklore Text Corpus

License: CC0-1.0

Locale: ky

Task: NLP

Format: TXT

Size: 1.28 MB

This dataset is a longitudinal news corpus for the Persian language sourced from Radio Farda from 2001 to 2026. It contains over 350,000 articles (51M tokens).

RFE/RL Persian News Text Corpus

License: CC-BY-NC-SA-4.0

Locale: fa

Task: NLP

Format: TXT

Size: 307.78 MB

Longitudinal Crimean Tatar news corpus from Qırım.Aqiqat (2014-2026) with over 32,000 articles and 7.5M tokens.

RFE/RL Crimean Tatar News Text Corpus

License: CC-BY-NC-SA-4.0

Locale: crh

Task: NLP

Format: TXT

Size: 18.35 MB

Collaborative Action For Research & Development (CARD)

IBT Torwali Wordlist

The IBT Torwali Wordlist contains approximately 20,000 unique entries in Torwali (ISO 639-3: trw), an under-documented Indo-Aryan language spoken in northern Pakistan. The dataset comprises standardized lexical entries covering core vocabulary, function words, and culturally salient terms, with consistent orthography and normalization suitable for linguistic and computational use. Entries are aligned with English and Urdu glosses, and include part-of-speech tag.

License: CC-BY-SA-4.0

Locale: trw

Task: NLP

Format: CSV

Size: 312.87 KB

This Mandar language dataset is text and audio data paired for languange documentation and linguistic research in technology.

Mandar Balanipa Speech Corpus

License: CC-BY-NC-4.0

Locale: mdr

Task: TTS

Format: WEBM, TSV

Size: 281.70 MB

A 74.4M-word, 267,887-article Urdu news corpus from Associated Press Pakistan (2015-2026), released for non-commercial research with APP permission.

Associated Press Pakistan 74.4 Million Urdu Corpus

License: CC-BY-NC-SA-4.0

Locale: urd

Task: NLP

Format: TXT

Size: 143.39 MB

A Ormuri dictionary by Rozi Khan Burki documenting Ormuri vocabulary and meanings to support language preservation, linguistic research, and NLP development.

Ormuri Bilingual Dictionary by Rozi Khan Burki

License: CC-BY-ND-4.0

Locale: oru

Task: TTS

Format: JPG, PDF, DOCX

Size: 15.68 MB

A 500K token corpus of Noakhalian linguistic data including drama scripts and cultural texts for professional NLP research and language preservation.

Noakhalian (নোয়াখাইল্লা) Text Corpus

License: CC-BY-NC-4.0

Locale: oak

Task: NLP

Format: TXT, DOCX

Size: 3.61 MB