Datasets | Mozilla Data Collective

Collaborative Action For Research & Development (CARD)

IBT Torwali Literature Corpus

A Torwali literature corpus (~233K tokens) covering poetry, folklore, biographies, and cultural texts for linguistic research and NLP development.

License: CC-BY-NC-4.0

Locale: trw

Task: NLP

Format: TXT

Size: 488.12 KB

Collaborative Action For Research & Development (CARD)

IBT Torwali Wordlist

The IBT Torwali Wordlist contains approximately 20,000 unique entries in Torwali (ISO 639-3: trw), an under-documented Indo-Aryan language spoken in northern Pakistan. The dataset comprises standardized lexical entries covering core vocabulary, function words, and culturally salient terms, with consistent orthography and normalization suitable for linguistic and computational use. Entries are aligned with English and Urdu glosses, and include part-of-speech tag.

License: CC-BY-SA-4.0

Locale: trw

Task: NLP

Format: CSV

Size: 312.87 KB

Common Voice

Common Voice Scripted Speech 26.0 - Torwali

A collection of read speech recordings in Torwali.

License: CC0-1.0

Locale: trw

Task: ASR

Format: MP3

Size: 374.55 MB

Forum for Language Initiatives

Khowar Literature Corpus by FLI

A multi-genre Khowar language corpus designed for linguistic research, NLP applications, and cultural documentation.

License: CC-BY-NC-4.0

Locale: khw

Task: NLP

Format: TXT

Size: 244.85 KB

MDC DatasetsCompensated

🚧 Khowar-English Parallel Corpus 🚧

A parallel corpus of over 10,000 sentences of Khowar and English. The sentences are derived from a multimodal corpus consisting of 152 transcribed and translated Khowar language videos.

License: MDC DLA-1.0

Locale: khw, en

Task: MT

Format: TSV

Size: 407.57 KB

Balochistan Educational and Cultural Organization

Talar (تلار) Barahui Magazine Corpus

A ~150,000-word Brahui corpus from the monthly magazine Talar, covering editorials, essays, fiction, poetry, and socio-cultural commentary.

License: CC-BY-NC-SA-4.0

Locale: brh

Task: NLP

Format: TXT

Size: 317.22 KB

Balochi Academy

Balochi Academy Text Corpus

This corpus contains approximately 500k tokens of text from novels, poetry, articles, riddles, and proverbs, covering both literary and traditional genres. It is intended for linguistic research, NLP tasks (e.g., language modeling and text analysis), and cultural documentation.

License: CC-BY-NC-SA-4.0

Locale: bgn

Task: NLP

Format: TXT

Size: 1.88 MB

MirasAI

Ormuri Literature corpus by Rozi Burki

Ormuri Literature Corpus is a collection of Ormuri literary texts by Rozi Burki, supporting research and preservation of this under-resourced language.

License: CC-BY-NC-SA-4.0

Locale: oru

Task: NLP

Format: PDF, DOCX

Size: 18.92 MB

Forum for Language Initiatives

Hussain Faizy Indus Kohistani Corpus

The Indus Kohistani corpus contains around 500k tokens of folktales, stories, poetry, biographies, and conversational texts, all transcribed with a consistent community orthography. Reviewed by native speakers, the corpus offers a representative snapshot of the language’s vocabulary and grammar for linguistic and computational research.

License: CC-BY-SA-4.0

Locale: mvy

Task: NLP

Format: TXT

Size: 14.70 MB

MirasAI

Wakhi Literature Corpus

A literature corpus of Wakhi language texts, preserving the language’s literary, cultural, and linguistic heritage for research and low-resource NLP.

License: CC-BY-ND-4.0

Locale: wbl

Task: NLP

Format: DOCX, TXT

Size: 74.83 KB

Balochi Academy

Eastern Balochi Literature Corpus

A UTF-8 normalized Eastern Balochi literature corpus (~1.9M tokens) covering poetry, folklore, novels, and cultural texts for linguistic research and NLP.

License: CC-BY-NC-4.0

Locale: bgp

Task: NLP

Format: TXT

Size: 949.67 KB

MEDIAMEN

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

This parallel sentences corpus containing 30,405 aligned sentence pairs with a total of approximately 0.62 million tokens, curated from the archival materials of Mediamen (Advertising Agency). The sentences were professionally translated from English into Punjabi (Shahmukhi) and are intended to support machine translation, linguistic research, and Punjabi language technology development, particularly for real-world and contemporary language use.

License: CC-BY-NC-4.0

Locale: en-PK, pnb

Task: MT

Format: CSV

Size: 1.08 MB

MirasAI

Rangpuri (অংপুরি Ôṅgpuri) Text Corpus

A 500K token corpus of Rangpuri literature and drama provided in UTF-8 format for linguistic research and NLP development.

License: CC-BY-NC-4.0

Locale: rkt

Task: NLP

Format: TXT

Size: 3.68 MB

MEDIAMEN

Mediamen Punjabi Literature Corpus

This corpus is a collection of one million tokens of Western Punjabi language. The data was produced under the Mediamen publishing agency over the last ten years. The corpus contains work of literature including short stories, novels, fiction, non-fiction, and dramas. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: pnb

Task: NLP

Format: TXT

Size: 1.82 MB

MirasAI

Sylheti Text Corpus by Haque Publishers

A 524K token Sylheti corpus featuring drama scripts and cultural texts in UTF-8 format for linguistic research and NLP development.

License: CC-BY-NC-4.0

Locale: syl

Task: NLP

Format: TXT

Size: 3.53 MB

Tamahi Suneha Magazine

Punjabi Literature Corpus

This corpus contains 10,39,430 tokens of Punjabi Shahmukhi script.

License: CC-BY-NC-4.0

Locale: pa-PK

Task: OTH

Format: TXT

Size: 1.83 MB

Chishti Sons

Chishti Sons Punjabi Literature Corpus

This corpus is a collection of more than one million tokens of Western Punjabi language. The data was produced under the Chishti Sons publishing agency. The corpus contains work of literature including short stories, stories, fiction, non-fiction, and other literary books. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: pnb

Task: NLP

Format: TXT

Size: 1.65 MB

MirasAI

Telugu Text Corpus

A Telugu text corpus for NLP research, containing diverse written content for language modeling, text analysis, and linguistic studies.

License: CC-BY-NC-SA-4.0

Locale: tel

Task: NLP

Format: TXT

Size: 19.87 MB

Community

Hindi 10 Million Text Corpus

Hindi monolingual corpus of about 10 million tokens, compiled from multiple authors and including literary works and personal articles.

License: CC-BY-ND-4.0

Locale: hin

Task: NLP

Format: DOCX

Size: 21.91 MB

Balochistan Educational and Cultural Organization

Western Balochi Literature Cropus

A cleaned literary corpus in Western Balochi (Rakhshani) including articles, research, translations, and creative literature. (~1.1 m tokens)

License: CC-BY-NC-4.0

Locale: bgn

Task: NLP

Format: TXT

Size: 2.26 MB

Forum for Language Initiatives

Khowar Word List

A UTF-8 encoded Khowar lexical corpus containing alphabet definitions and a structured word list for linguistic research and language technology development.

License: CC-BY-NC-4.0

Locale: khw

Task: NLP

Format: TXT

Size: 64.22 KB

Kaleem Art Press

Saraiki-English Parallel Corpus

English–Saraiki parallel corpus: 51,447 aligned sentence pairs (~0.89M words), translated by Kaleem Art Press for MT and Saraiki NLP research.

License: CC-BY-NC-4.0

Locale: mul

Task: MT

Format: CSV

Size: 1.92 MB

MDC DatasetsCompensated

🚧 Khowar Speech Translation Corpus 🚧

A Khowar-English speech translation dataset with approximately 9.5 hours of Khowar audio with English text translations.

License: MDC DLA-1.0

Locale: khw, en

Task: MT

Format: MP3, TSV

Size: 244.96 MB

Baloch Publishers Multan

Baloch Publishers Saraiki Literature Corpus

This corpus is a collection of one million tokens of Saraiki language. The data was produced under the Baloch Publishers over the last ten years. The corpus contains work of literature including dictionaries, short stories, novels, fiction, non-fiction, and travelogue. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: skr

Task: NLP

Format: TXT

Size: 2.04 MB

All Datasets

Not sure where to start?

IBT Torwali Literature Corpus

IBT Torwali Wordlist

Common Voice Scripted Speech 26.0 - Torwali

Khowar Literature Corpus by FLI

🚧 Khowar-English Parallel Corpus 🚧

Talar (تلار) Barahui Magazine Corpus

Balochi Academy Text Corpus

Ormuri Literature corpus by Rozi Burki

Hussain Faizy Indus Kohistani Corpus

Wakhi Literature Corpus

Eastern Balochi Literature Corpus

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

Rangpuri (অংপুরি Ôṅgpuri) Text Corpus

Mediamen Punjabi Literature Corpus

Sylheti Text Corpus by Haque Publishers

Punjabi Literature Corpus

Chishti Sons Punjabi Literature Corpus

Telugu Text Corpus

Hindi 10 Million Text Corpus

Western Balochi Literature Cropus

Khowar Word List

Saraiki-English Parallel Corpus

🚧 Khowar Speech Translation Corpus 🚧

Baloch Publishers Saraiki Literature Corpus