Datasets | Mozilla Data Collective

Forum for Language Initiatives

Kohistani Shina Word List

A UTF-8 encoded Kohistani Shina dictionary and word list corpus with lexical entries, meanings, and annotations for linguistic research and NLP use.

License: CC-BY-NC-4.0

Locale: plk

Task: NLP

Format: TXT

Size: 394.05 KB

Common Voice

Common Voice Scripted Speech 26.0 - Kohistani Shina

A collection of read speech recordings in Kohistani Shina.

License: CC0-1.0

Locale: plk

Task: ASR

Format: MP3

Size: 330.88 MB

Forum for Language Initiatives

Khowar Word List

A UTF-8 encoded Khowar lexical corpus containing alphabet definitions and a structured word list for linguistic research and language technology development.

License: CC-BY-NC-4.0

Locale: khw

Task: NLP

Format: TXT

Size: 64.22 KB

Forum for Language Initiatives

Hussain Faizy Indus Kohistani Corpus

The Indus Kohistani corpus contains around 500k tokens of folktales, stories, poetry, biographies, and conversational texts, all transcribed with a consistent community orthography. Reviewed by native speakers, the corpus offers a representative snapshot of the language’s vocabulary and grammar for linguistic and computational research.

License: CC-BY-SA-4.0

Locale: mvy

Task: NLP

Format: TXT

Size: 14.70 MB

MEDIAMEN

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

This parallel sentences corpus containing 30,405 aligned sentence pairs with a total of approximately 0.62 million tokens, curated from the archival materials of Mediamen (Advertising Agency). The sentences were professionally translated from English into Punjabi (Shahmukhi) and are intended to support machine translation, linguistic research, and Punjabi language technology development, particularly for real-world and contemporary language use.

License: CC-BY-NC-4.0

Locale: en-PK, pnb

Task: MT

Format: CSV

Size: 1.08 MB

Common Voice

Common Voice Scripted Speech 26.0 - Indus Kohistani

A collection of read speech recordings in Indus Kohistani.

License: CC0-1.0

Locale: mvy

Task: ASR

Format: MP3

Size: 448.96 MB

Chishti Sons

Chishti Sons Punjabi Literature Corpus

This corpus is a collection of more than one million tokens of Western Punjabi language. The data was produced under the Chishti Sons publishing agency. The corpus contains work of literature including short stories, stories, fiction, non-fiction, and other literary books. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: pnb

Task: NLP

Format: TXT

Size: 1.65 MB

Community

Sindhi Word Segmentation (SdSEG)

A sentence-level annotated dataset for training and evaluating sequence labeling models such as CRF, BiLSTM, and transformer-based architectures.

License: CC-BY-NC-4.0

Locale: snd

Task: NLP

Format: JSON

Size: 1.13 MB

Collaborative Action For Research & Development (CARD)

IBT Torwali Wordlist

The IBT Torwali Wordlist contains approximately 20,000 unique entries in Torwali (ISO 639-3: trw), an under-documented Indo-Aryan language spoken in northern Pakistan. The dataset comprises standardized lexical entries covering core vocabulary, function words, and culturally salient terms, with consistent orthography and normalization suitable for linguistic and computational use. Entries are aligned with English and Urdu glosses, and include part-of-speech tag.

License: CC-BY-SA-4.0

Locale: trw

Task: NLP

Format: CSV

Size: 312.87 KB

MDC DatasetsCompensated

🚧 Khowar-English Parallel Corpus 🚧

A parallel corpus of over 10,000 sentences of Khowar and English. The sentences are derived from a multimodal corpus consisting of 152 transcribed and translated Khowar language videos.

License: MDC DLA-1.0

Locale: khw, en

Task: MT

Format: TSV

Size: 407.57 KB

Balochistan Educational and Cultural Organization

Western Balochi Literature Cropus

A cleaned literary corpus in Western Balochi (Rakhshani) including articles, research, translations, and creative literature. (~1.1 m tokens)

License: CC-BY-NC-4.0

Locale: bgn

Task: NLP

Format: TXT

Size: 2.26 MB

MirasAI

Ormuri Bilingual Dictionary by Rozi Khan Burki

A Ormuri dictionary by Rozi Khan Burki documenting Ormuri vocabulary and meanings to support language preservation, linguistic research, and NLP development.

License: CC-BY-ND-4.0

Locale: oru

Task: TTS

Format: JPG, PDF, DOCX

Size: 15.68 MB

Community

Sindhi Parts Of Speech (SiPOS)

A large-scale Sindhi Part-of-Speech (SiPOS) tagging dataset for developing and evaluating NLP models in a low-resource language.

License: CC-BY-NC-4.0

Locale: snd

Task: NLP

Format: TXT

Size: 874.04 KB

Baloch Publishers Multan

Baloch Publishers Saraiki Literature Corpus

This corpus is a collection of one million tokens of Saraiki language. The data was produced under the Baloch Publishers over the last ten years. The corpus contains work of literature including dictionaries, short stories, novels, fiction, non-fiction, and travelogue. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: skr

Task: NLP

Format: TXT

Size: 2.04 MB

MEDIAMEN

Mediamen Punjabi Literature Corpus

This corpus is a collection of one million tokens of Western Punjabi language. The data was produced under the Mediamen publishing agency over the last ten years. The corpus contains work of literature including short stories, novels, fiction, non-fiction, and dramas. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: pnb

Task: NLP

Format: TXT

Size: 1.82 MB

Common Voice

Common Voice Scripted Speech 26.0 - Shina

A collection of read speech recordings in Shina.

License: CC0-1.0

Locale: scl

Task: ASR

Format: MP3

Size: 204.73 MB

Balochistan Educational and Cultural Organization

NAWA-E-WATAN Balochi Newspaper Corpus

A ~1.02M-token Balochi newspaper corpus from NAWA-E-WATAN, representing contemporary journalistic and public discourse.

License: CC-BY-NC-4.0

Locale: bgn

Task: NLP

Format: TXT

Size: 1.43 MB

Common Voice

Common Voice Scripted Speech 26.0 - Khetrani

A collection of read speech recordings in Khetrani.

License: CC0-1.0

Locale: xhe

Task: ASR

Format: MP3

Size: 196.18 MB

Forum for Language Initiatives

Khowar Literature Corpus by FLI

A multi-genre Khowar language corpus designed for linguistic research, NLP applications, and cultural documentation.

License: CC-BY-NC-4.0

Locale: khw

Task: NLP

Format: TXT

Size: 244.85 KB

Tamahi Suneha Magazine

Punjabi Literature Corpus

This corpus contains 10,39,430 tokens of Punjabi Shahmukhi script.

License: CC-BY-NC-4.0

Locale: pa-PK

Task: OTH

Format: TXT

Size: 1.83 MB

Kaleem Art Press

Jhoke Publisher Multan’s Saraiki Newspaper Corpus

Jhoke Publisher Multan’s Saraiki Newspaper Corpus (~1.25M tokens) is a normalized UTF-8 collection of Saraiki newspaper.

License: CC-BY-NC-4.0

Locale: skr

Task: NLP

Format: TXT

Size: 2.30 MB

Common Voice

Common Voice Scripted Speech 26.0 - Loja Highland Kichwa

A collection of read speech recordings in Loja Highland Kichwa.

License: CC0-1.0

Locale: qvj

Task: ASR

Format: MP3

Size: 221.76 MB

PROXIMA

Sindhi Transliteration Dataset by Proxima AI

Sindhi transliteration dataset containing Sindhi sentences paired with Roman Sindhi transliterations for NLP research and model development.

License: CC-BY-NC-SA-4.0

Locale: snd

Task: NLP

Format: CSV

Size: 5.09 MB

Common Voice

Common Voice Scripted Speech 26.0 - Eastern Balochi

A collection of read speech recordings in Eastern Balochi.

License: CC0-1.0

Locale: bgp

Task: ASR

Format: MP3

Size: 259.69 MB

All Datasets

Not sure where to start?

Kohistani Shina Word List

Common Voice Scripted Speech 26.0 - Kohistani Shina

Khowar Word List

Hussain Faizy Indus Kohistani Corpus

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

Common Voice Scripted Speech 26.0 - Indus Kohistani

Chishti Sons Punjabi Literature Corpus

Sindhi Word Segmentation (SdSEG)

IBT Torwali Wordlist

🚧 Khowar-English Parallel Corpus 🚧

Western Balochi Literature Cropus

Ormuri Bilingual Dictionary by Rozi Khan Burki

Sindhi Parts Of Speech (SiPOS)

Baloch Publishers Saraiki Literature Corpus

Mediamen Punjabi Literature Corpus

Common Voice Scripted Speech 26.0 - Shina

NAWA-E-WATAN Balochi Newspaper Corpus

Common Voice Scripted Speech 26.0 - Khetrani

Khowar Literature Corpus by FLI

Punjabi Literature Corpus

Jhoke Publisher Multan’s Saraiki Newspaper Corpus

Common Voice Scripted Speech 26.0 - Loja Highland Kichwa

Sindhi Transliteration Dataset by Proxima AI

Common Voice Scripted Speech 26.0 - Eastern Balochi