Datasets | Mozilla Data Collective

Collaborative Action For Research & Development (CARD)

IBT Torwali Wordlist

The IBT Torwali Wordlist contains approximately 20,000 unique entries in Torwali (ISO 639-3: trw), an under-documented Indo-Aryan language spoken in northern Pakistan. The dataset comprises standardized lexical entries covering core vocabulary, function words, and culturally salient terms, with consistent orthography and normalization suitable for linguistic and computational use. Entries are aligned with English and Urdu glosses, and include part-of-speech tag.

License: CC-BY-SA-4.0

Locale: trw

Task: NLP

Format: CSV

Size: 312.87 KB

Collaborative Action For Research & Development (CARD)

IBT Torwali Literature Corpus

A Torwali literature corpus (~233K tokens) covering poetry, folklore, biographies, and cultural texts for linguistic research and NLP development.

License: CC-BY-NC-4.0

Locale: trw

Task: NLP

Format: TXT

Size: 488.12 KB

Common Voice

Common Voice Scripted Speech 26.0 - Torwali

A collection of read speech recordings in Torwali.

License: CC0-1.0

Locale: trw

Task: ASR

Format: MP3

Size: 374.55 MB

Forum for Language Initiatives

Khowar Word List

A UTF-8 encoded Khowar lexical corpus containing alphabet definitions and a structured word list for linguistic research and language technology development.

License: CC-BY-NC-4.0

Locale: khw

Task: NLP

Format: TXT

Size: 64.22 KB

Forum for Language Initiatives

Kohistani Shina Word List

A UTF-8 encoded Kohistani Shina dictionary and word list corpus with lexical entries, meanings, and annotations for linguistic research and NLP use.

License: CC-BY-NC-4.0

Locale: plk

Task: NLP

Format: TXT

Size: 394.05 KB

MirasAI

Ormuri Bilingual Dictionary by Rozi Khan Burki

A Ormuri dictionary by Rozi Khan Burki documenting Ormuri vocabulary and meanings to support language preservation, linguistic research, and NLP development.

License: CC-BY-ND-4.0

Locale: oru

Task: TTS

Format: JPG, PDF, DOCX

Size: 15.68 MB

Balochistan Educational and Cultural Organization

Talar (تلار) Barahui Magazine Corpus

A ~150,000-word Brahui corpus from the monthly magazine Talar, covering editorials, essays, fiction, poetry, and socio-cultural commentary.

License: CC-BY-NC-SA-4.0

Locale: brh

Task: NLP

Format: TXT

Size: 317.22 KB

MEDIAMEN

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

This parallel sentences corpus containing 30,405 aligned sentence pairs with a total of approximately 0.62 million tokens, curated from the archival materials of Mediamen (Advertising Agency). The sentences were professionally translated from English into Punjabi (Shahmukhi) and are intended to support machine translation, linguistic research, and Punjabi language technology development, particularly for real-world and contemporary language use.

License: CC-BY-NC-4.0

Locale: en-PK, pnb

Task: MT

Format: CSV

Size: 1.08 MB

Balochi Academy

Balochi Academy Text Corpus

This corpus contains approximately 500k tokens of text from novels, poetry, articles, riddles, and proverbs, covering both literary and traditional genres. It is intended for linguistic research, NLP tasks (e.g., language modeling and text analysis), and cultural documentation.

License: CC-BY-NC-SA-4.0

Locale: bgn

Task: NLP

Format: TXT

Size: 1.88 MB

Community

Oro_Word

Afaan Oromoo word-level speech dataset collected to support open-source speech recognition and text-to-speech technology.

License: CC0-1.0

Locale: om

Task: TTS

Format: .WAV, CSV

Size: 1.28 MB

MDC DatasetsCompensated

🚧 Khowar-English Parallel Corpus 🚧

A parallel corpus of over 10,000 sentences of Khowar and English. The sentences are derived from a multimodal corpus consisting of 152 transcribed and translated Khowar language videos.

License: MDC DLA-1.0

Locale: khw, en

Task: MT

Format: TSV

Size: 407.57 KB

Forum for Language Initiatives

Hussain Faizy Indus Kohistani Corpus

The Indus Kohistani corpus contains around 500k tokens of folktales, stories, poetry, biographies, and conversational texts, all transcribed with a consistent community orthography. Reviewed by native speakers, the corpus offers a representative snapshot of the language’s vocabulary and grammar for linguistic and computational research.

License: CC-BY-SA-4.0

Locale: mvy

Task: NLP

Format: TXT

Size: 14.70 MB

Community

Sindhi Word Segmentation (SdSEG)

A sentence-level annotated dataset for training and evaluating sequence labeling models such as CRF, BiLSTM, and transformer-based architectures.

License: CC-BY-NC-4.0

Locale: snd

Task: NLP

Format: JSON

Size: 1.13 MB

Institute of African Digital Humanities

Teke-Laali-TTS-Dataset

The dataset contains paired audio and text resources for Teke-Laali, a Bantu language spoken in the Congo. It consists of seven folders containing a total of 9,069 audio clips from raw audio recordings, with a total duration of 7:01:50.126 (HH:MM:SS.mmm). Additionally, there are seven audio/text mapping files containing a total of 9,069 lines. The dataset is suitable for TTS tasks.

License: NOODL-1.0

Locale: lli

Task: TTS

Format: WAV, TSV

Size: 635.61 MB

Balochistan Educational and Cultural Organization

NAWA-E-WATAN Balochi Newspaper Corpus

A ~1.02M-token Balochi newspaper corpus from NAWA-E-WATAN, representing contemporary journalistic and public discourse.

License: CC-BY-NC-4.0

Locale: bgn

Task: NLP

Format: TXT

Size: 1.43 MB

MirasAI

Ormuri Literature corpus by Rozi Burki

Ormuri Literature Corpus is a collection of Ormuri literary texts by Rozi Burki, supporting research and preservation of this under-resourced language.

License: CC-BY-NC-SA-4.0

Locale: oru

Task: NLP

Format: PDF, DOCX

Size: 18.92 MB

Institute of African Digital Humanities

Kituba-TTS-Dataset

Paired audio and text data on Kituba (mkw), a language spoken in Congo. The audio corpus consists of 8,302 clips read by one speaker, totalling 350 min 11.98 sec. The dataset also contains a mapping file of audio and text with 8,173 lines. Each line begins with the name of an audio file, followed by a tab and then the corresponding text excerpt. This dataset is suitable for TTS tasks.

License: NOODL-1.0

Locale: mkw

Task: TTS

Format: WAV, TSV

Size: 553.28 MB

CLEAR Global

TWB Parallel Sentence kits - Kanuri (5k)

5,000 English–Kanuri parallel sentences from CLEAR Global's Gamayun mini-kit, translated by professional translators from Tatoeba source sentences.

License: CC-BY-4.0

Locale: kau

Task: MT

Format: TSV

Size: 358.46 KB

CLEAR Global

TWB Parallel Sentence kits - Tigrinya (5k)

5,000 English–Tigrinya parallel sentences from CLEAR Global's Gamayun mini-kit, translated by professional translators from Tatoeba source sentences.

License: CC-BY-4.0

Locale: tig

Task: MT

Format: TSV

Size: 404.75 KB

Common Voice

Common Voice Scripted Speech 26.0 - Gawarbaiti

A collection of read speech recordings in Gawarbaiti.

License: CC0-1.0

Locale: gwt

Task: ASR

Format: MP3

Size: 249.85 MB

Tamazight NLP

AmaWar

Amawal Warayni - ⴰⵎⴰⵡⴰⵍ ⴰⵙⵏⵎⴰⵍⴰⵢ ⵏ ⵉⵏⵓⵎⴰⴽ ⵏ ⵡⴰⵙⵙⴰⵖⵏ ⴷ ⵉⵎⵢⴰⴳⵏ ⵏ ⵜⵎⴰⵣⵉⵖⵜ ⵏ ⴰⵢⵜ ⵡⴰⵔⴰⵢⵏ

License: Apache-2.0

Locale: rif

Task: MT

Format: TSV

Size: 100.28 KB

Community

Hindi 10 Million Text Corpus

Hindi monolingual corpus of about 10 million tokens, compiled from multiple authors and including literary works and personal articles.

License: CC-BY-ND-4.0

Locale: hin

Task: NLP

Format: DOCX

Size: 21.91 MB

Common Voice

Common Voice Scripted Speech 26.0 - Dhatki

A collection of read speech recordings in Dhatki.

License: CC0-1.0

Locale: mki

Task: ASR

Format: MP3

Size: 187.83 MB

MEDIAMEN

Mediamen Punjabi Literature Corpus

This corpus is a collection of one million tokens of Western Punjabi language. The data was produced under the Mediamen publishing agency over the last ten years. The corpus contains work of literature including short stories, novels, fiction, non-fiction, and dramas. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: pnb

Task: NLP

Format: TXT

Size: 1.82 MB

All Datasets

Not sure where to start?

IBT Torwali Wordlist

IBT Torwali Literature Corpus

Common Voice Scripted Speech 26.0 - Torwali

Khowar Word List

Kohistani Shina Word List

Ormuri Bilingual Dictionary by Rozi Khan Burki

Talar (تلار) Barahui Magazine Corpus

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

Balochi Academy Text Corpus

Oro_Word

🚧 Khowar-English Parallel Corpus 🚧

Hussain Faizy Indus Kohistani Corpus

Sindhi Word Segmentation (SdSEG)

Teke-Laali-TTS-Dataset

NAWA-E-WATAN Balochi Newspaper Corpus

Ormuri Literature corpus by Rozi Burki

Kituba-TTS-Dataset

TWB Parallel Sentence kits - Kanuri (5k)

TWB Parallel Sentence kits - Tigrinya (5k)

Common Voice Scripted Speech 26.0 - Gawarbaiti

AmaWar

Hindi 10 Million Text Corpus

Common Voice Scripted Speech 26.0 - Dhatki

Mediamen Punjabi Literature Corpus