Datasets | Mozilla Data Collective

A 500K token corpus of Rangpuri literature and drama provided in UTF-8 format for linguistic research and NLP development.

Rangpuri (অংপুরি Ôṅgpuri) Text Corpus

License: CC-BY-NC-4.0

Locale: rkt

Task: NLP

Format: TXT

Size: 3.68 MB

MDC DatasetsCompensated

🚧 Khowar-English Parallel Corpus 🚧

A parallel corpus of over 10,000 sentences of Khowar and English. The sentences are derived from a multimodal corpus consisting of 152 transcribed and translated Khowar language videos.

License: MDC DLA-1.0

Locale: khw, en

Task: MT

Format: TSV

Size: 407.57 KB

A Ormuri dictionary by Rozi Khan Burki documenting Ormuri vocabulary and meanings to support language preservation, linguistic research, and NLP development.

Ormuri Bilingual Dictionary by Rozi Khan Burki

License: CC-BY-ND-4.0

Locale: oru

Task: TTS

Format: JPG, PDF, DOCX

Size: 15.68 MB

A 500K token corpus of Noakhalian linguistic data including drama scripts and cultural texts for professional NLP research and language preservation.

Noakhalian (নোয়াখাইল্লা) Text Corpus

License: CC-BY-NC-4.0

Locale: oak

Task: NLP

Format: TXT, DOCX

Size: 3.61 MB

A 74.4M-word, 267,887-article Urdu news corpus from Associated Press Pakistan (2015-2026), released for non-commercial research with APP permission.

Associated Press Pakistan 74.4 Million Urdu Corpus

License: CC-BY-NC-SA-4.0

Locale: urd

Task: NLP

Format: TXT

Size: 143.39 MB

SdCorpus is a large scale monolingual Sindhi text corpus collected from publicly available web sources for NLP and language model development.

Sindhi Corpus

License: CC-BY-NC-4.0

Locale: snd

Task: LM

Format: TXT

Size: 321.00 MB

A 15.6k-word corpus of 19 Dolgan fairy tales, digitized to catalyze NLP research and revitalization for this highly endangered Siberian Turkic language.

Dolgan Folklore Text Corpus

License: CC0-1.0

Locale: dlg

Task: NLP

Format: TXT

Size: 57.16 KB

A 16.9M-word Balochi news corpus from Associated Press of Pakistan, covering 59,172 articles published between 2016 and 2026.

Associated Press Pakistan 16.9 Million Balochi Corpus

License: CC-BY-NC-SA-4.0

Locale: bal

Task: NLP

Format: TXT

Size: 22.85 MB

A 427k-word Kyrgyz folklore corpus of tales, proverbs, and aphorisms, digitized from 5 Bishkek academic volumes (2016-2017) for NLP tasks.

Kyrgyz Folklore Text Corpus

License: CC0-1.0

Locale: ky

Task: NLP

Format: TXT

Size: 1.28 MB

This dataset is a longitudinal news corpus for the Pashto language, sourced from Radio Azadi, from 2000 to 2026, it contains over 204,000 articles (16M tokens).

RFE/RL Afghan Pashto News Text Corpus

License: CC-BY-NC-SA-4.0

Locale: ps

Task: NLP

Format: TXT

Size: 117.68 MB

Forum for Language Initiatives

Khowar Word List

A UTF-8 encoded Khowar lexical corpus containing alphabet definitions and a structured word list for linguistic research and language technology development.

License: CC-BY-NC-4.0

Locale: khw

Task: NLP

Format: TXT

Size: 64.22 KB

TAUSCompensated

🚧 Colloquial Parallel Text (English - Hausa) 🚧

49k native-reviewed colloquial sentences translated from US English into Hausa for fine-tuning conversational MT and AI models.

License: TAUS DUA

Locale: en-US, ha-NG

Task: MT

Format: TSV

Size: 2.93 MB

A collection of annotated images capturing public space activities, traditional handicrafts, historical and cultural heritage exhibits in Java.

IND IKRAR - Craft and Cultural Heritage Multimodal Corpus

License: CC-BY-NC-SA-4.0

Locale: ind

Task: CV

Format: JPEG, WEBM, JSON

Size: 286.27 MB

TAUSCompensated

🚧 Colloquial Parallel Text (English - Hausa) 🚧

100k native-reviewed colloquial sentences translated from US English into Hausa for fine-tuning conversational MT and AI models.

License: TAUS DUA

Locale: en-US, ha-NG

Task: MT

Format: TSV

Size: 4.49 MB

A 4.2-million-word corpus of 54 iconic Polish novels, multi-volume epics, and documentary prose pieces from the late 19th and early 20th centuries.

Polish Public Domain 20th Century Literature Text Corpus

License: CC0-1.0

Locale: pl

Task: NLP

Format: TXT

Size: 10.86 MB

A corpus of traditional Uzbek proverbs digitized from a 2005 volume, designed for NLP tasks and LLM cultural competence evaluations.

Uzbek Proverbs Text Corpus

License: CC0-1.0

Locale: uz

Task: NLP

Format: TXT

Size: 142.68 KB

Longitudinal Pakistani Pashto news corpus from Radio Mashaal (2010-2026) with over 52,000 articles and 14.5M tokens.

RFE/RL Pashto (Pakistani) News Text Corpus

License: CC-BY-NC-SA-4.0

Locale: ps

Task: NLP

Format: TXT

Size: 39.26 MB

MDC DatasetsCompensated

🚧 Hazargi-English Parallel Text Corpus 🚧

A Hazargi-English parallel corpus derived from transcribed and translated video recordings

License: MDC DLA-1.0

Locale: haz, en

Task: MT

Format: TSV

Size: 434.04 KB

Text corpus of 176,843 words compiled from 397 Judeo-Espanyol articles from Şalom newspaper.

Şalom Ladino Corpus

License: CC-BY-4.0

Locale: lad

Task: LM

Format: TXT

Size: 403.16 KB

A 690K token corpus of raw Chittagonian text covering drama, poetry, and folklore for linguistic research and NLP development.

Chittagonian (চাটগাঁইয়া, saṭgãia) Text Corpus

License: CC-BY-NC-SA-4.0

Locale: ctg

Task: NLP

Format: TXT

Size: 6.59 MB

This dataset is longitudinal news corpus for the Tatar and Bashkir languages, sourced from Azatliq Radiosi, from 2001 to 2025, it contains over 105,000 articles (30M tokens).

RFE/RL Tatar-Bashkir News Text Corpus

License: CC-BY-NC-SA-4.0

Locale: tt,ba,ru

Task: NLP

Format: TXT

Size: 102.44 MB

A collection of annotated images capturing public space activities, traditional handicrafts, historical and cultural heritage exhibits in Java.

JAV IKRAR - Craft and Cultural Heritage Multimodal Corpus

License: CC-BY-NC-SA-4.0

Locale: jav

Task: CV

Format: JPEG, WEBM, JSON

Size: 1.13 GB

LocaleNLP

English Hausa Parallel Corpus

An English–Hausa dataset with 5,000 sentence pairs useful for machine translation and basic language processing tasks.

License: CC-BY-NC-4.0

Locale: eng, hau

Task: MT

Format: csv

Size: 164.32 KB