Datasets | Mozilla Data Collective

MEDIAMEN

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

This parallel sentences corpus containing 30,405 aligned sentence pairs with a total of approximately 0.62 million tokens, curated from the archival materials of Mediamen (Advertising Agency). The sentences were professionally translated from English into Punjabi (Shahmukhi) and are intended to support machine translation, linguistic research, and Punjabi language technology development, particularly for real-world and contemporary language use.

License: CC-BY-NC-4.0

Locale: en-PK, pnb

Task: MT

Format: CSV

Size: 1.08 MB

MEDIAMEN

Mediamen Punjabi Literature Corpus

This corpus is a collection of one million tokens of Western Punjabi language. The data was produced under the Mediamen publishing agency over the last ten years. The corpus contains work of literature including short stories, novels, fiction, non-fiction, and dramas. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: pnb

Task: NLP

Format: TXT

Size: 1.82 MB

Chishti Sons

Chishti Sons Punjabi Literature Corpus

This corpus is a collection of more than one million tokens of Western Punjabi language. The data was produced under the Chishti Sons publishing agency. The corpus contains work of literature including short stories, stories, fiction, non-fiction, and other literary books. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: pnb

Task: NLP

Format: TXT

Size: 1.65 MB

Common Voice

Common Voice Scripted Speech 26.0 - Punjabi

A collection of read speech recordings in Punjabi.

License: CC0-1.0

Locale: pa-IN

Task: ASR

Format: MP3

Size: 104.82 MB

MirasAI

Punjabi 10 Hours TTS

10-hour Punjabi TTS dataset in Shahmukhi script with paired audio and transcripts, designed for speech synthesis and Punjabi language technology.

License: CC-BY-NC-SA-4.0

Locale: pnb

Task: TTS

Format: WEBM, TSV

Size: 481.96 MB

Tamahi Suneha Magazine

Punjabi Literature Corpus

This corpus contains 10,39,430 tokens of Punjabi Shahmukhi script.

License: CC-BY-NC-4.0

Locale: pa-PK

Task: OTH

Format: TXT

Size: 1.83 MB

MirasAI

Associated Press Pakistan 21.4 Million Pashto Corpus

Pashto news corpus from the Associated Press of Pakistan, containing 99,495 articles and 21.4 million words from 2015–2026.

License: CC-BY-NC-SA-4.0

Locale: pus

Task: NLP

Format: TXT

Size: 38.81 MB

PROXIMA

Synthetic Urdu 2

Synthetic Urdu sentences audio dataset of 85,327 verse style audio-text pairs.

License: CC-BY-NC-SA-4.0

Locale: urd

Task: TTS

Format: Parquet

Size: 14.80 GB

MDC DatasetsCompensated

🚧 Northern Pashto Multimodal Dataset 🚧

A Northern Pashto multimodal dataset with 10 hours of synchronized video, audio, transcriptions, speech segmentation, and ELAN annotations.

License: MDC DLA-1.0

Locale: pbu

Task: NLP

Format: MP4, WAV, EAF

Size: 251.41 GB

PROXIMA

PII Redact Urdu UOPF

Curated Urdu dataset of 100,000 records with fabricated PII spans, for training and evaluating privacy span detection across 12 domains.

License: CC-BY-NC-SA-4.0

Locale: urd

Task: NLP

Format: JSONL

Size: 5.77 MB

MDC DatasetsCompensated

Single Speaker Urdu TTS dataset (Female Speaker, South Punjab)

A high-quality single-speaker Urdu text-to-speech (TTS) dataset with 25 hours of paired speech recordings and transcriptions for speech synthesis.

License: MDC DLA-1.0

Locale: ur

Task: TTS

Format: WAV, TSV

Size: 6.02 GB

MirasAI

Associated Press Pakistan 74.4 Million Urdu Corpus

A 74.4M-word, 267,887-article Urdu news corpus from Associated Press Pakistan (2015-2026), released for non-commercial research with APP permission.

License: CC-BY-NC-SA-4.0

Locale: urd

Task: NLP

Format: TXT

Size: 143.39 MB

Rana Printers Multan

Rana Printers Urdu Literature Corpus

This corpus comprises 1.68 million tokens of high-quality Urdu text collected over the past decade through Rana Printers. It includes a diverse range of literary genres such as stories, short stories, novels, fiction, non-fiction, poetry, and historical works. All content is shared with the authors’ approval. The dataset is intended to support linguistic research, Urdu language technology development, and the preservation of literary and cultural heritage.

License: CC-BY-NC-4.0

Locale: ur

Task: OTH

Format: TXT

Size: 3.00 MB

Common Voice

Common Voice Scripted Speech 26.0 - Pashto

A collection of read speech recordings in Pashto.

License: CC0-1.0

Locale: ps

Task: ASR

Format: MP3

Size: 101.84 GB

Community

Sindhi Parts Of Speech (SiPOS)

A large-scale Sindhi Part-of-Speech (SiPOS) tagging dataset for developing and evaluating NLP models in a low-resource language.

License: CC-BY-NC-4.0

Locale: snd

Task: NLP

Format: TXT

Size: 874.04 KB

Community

Sindhi Word Segmentation (SdSEG)

A sentence-level annotated dataset for training and evaluating sequence labeling models such as CRF, BiLSTM, and transformer-based architectures.

License: CC-BY-NC-4.0

Locale: snd

Task: NLP

Format: JSON

Size: 1.13 MB

Forum for Language Initiatives

Kohistani Shina Word List

A UTF-8 encoded Kohistani Shina dictionary and word list corpus with lexical entries, meanings, and annotations for linguistic research and NLP use.

License: CC-BY-NC-4.0

Locale: plk

Task: NLP

Format: TXT

Size: 394.05 KB

Common Voice

Common Voice Spontaneous Speech 4.0 - Pashto

A collection of spontaneous responses to questions in Pashto.

License: CC0-1.0

Locale: ps

Task: ASR

Format: MP3

Size: 223.81 MB

Balochistan Educational and Cultural Organization

NAWA-E-WATAN Balochi Newspaper Corpus

A ~1.02M-token Balochi newspaper corpus from NAWA-E-WATAN, representing contemporary journalistic and public discourse.

License: CC-BY-NC-4.0

Locale: bgn

Task: NLP

Format: TXT

Size: 1.43 MB

MDC DatasetsCompensated

🚧 Khowar-English Parallel Corpus 🚧

A parallel corpus of over 10,000 sentences of Khowar and English. The sentences are derived from a multimodal corpus consisting of 152 transcribed and translated Khowar language videos.

License: MDC DLA-1.0

Locale: khw, en

Task: MT

Format: TSV

Size: 407.57 KB

PROXIMA

Urdu-Sindhi Grapheme-to-Phoneme Dataset

A bilingual Urdu and Sindhi grapheme-to-phoneme dataset for G2P, TTS, ASR, and speech technology research.

License: CC-BY-NC-SA-4.0

Locale: ur, sd

Task: TTS

Format: PARQUET

Size: 1.92 MB

Common Voice

Common Voice Scripted Speech 26.0 - Indus Kohistani

A collection of read speech recordings in Indus Kohistani.

License: CC0-1.0

Locale: mvy

Task: ASR

Format: MP3

Size: 448.96 MB

Community

Urdu Multi-Speaker TTS Dataset

An Urdu multi-speaker TTS dataset distributed in 36 zip files, each containing audio files and a TSV mapping file, with approximately 10 hours of speech.

License: CC-BY-NC-4.0

Locale: urd

Task: TTS

Format: WEBM, TSV

Size: 514.54 MB

Common Voice

Common Voice Scripted Speech 26.0 - Kohistani Shina

A collection of read speech recordings in Kohistani Shina.

License: CC0-1.0

Locale: plk

Task: ASR

Format: MP3

Size: 330.88 MB

All Datasets

Not sure where to start?

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

Mediamen Punjabi Literature Corpus

Chishti Sons Punjabi Literature Corpus

Common Voice Scripted Speech 26.0 - Punjabi

Punjabi 10 Hours TTS

Punjabi Literature Corpus

Associated Press Pakistan 21.4 Million Pashto Corpus

Synthetic Urdu 2

🚧 Northern Pashto Multimodal Dataset 🚧

PII Redact Urdu UOPF

Single Speaker Urdu TTS dataset (Female Speaker, South Punjab)

Associated Press Pakistan 74.4 Million Urdu Corpus

Rana Printers Urdu Literature Corpus

Common Voice Scripted Speech 26.0 - Pashto

Sindhi Parts Of Speech (SiPOS)

Sindhi Word Segmentation (SdSEG)

Kohistani Shina Word List

Common Voice Spontaneous Speech 4.0 - Pashto

NAWA-E-WATAN Balochi Newspaper Corpus

🚧 Khowar-English Parallel Corpus 🚧

Urdu-Sindhi Grapheme-to-Phoneme Dataset

Common Voice Scripted Speech 26.0 - Indus Kohistani

Urdu Multi-Speaker TTS Dataset

Common Voice Scripted Speech 26.0 - Kohistani Shina