Datasets | Mozilla Data Collective

MEDIAMEN

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

This parallel sentences corpus containing 30,405 aligned sentence pairs with a total of approximately 0.62 million tokens, curated from the archival materials of Mediamen (Advertising Agency). The sentences were professionally translated from English into Punjabi (Shahmukhi) and are intended to support machine translation, linguistic research, and Punjabi language technology development, particularly for real-world and contemporary language use.

License: CC-BY-NC-4.0

Locale: en-PK, pnb

Task: MT

Format: CSV

Size: 1.08 MB

MEDIAMEN

Mediamen Punjabi Literature Corpus

This corpus is a collection of one million tokens of Western Punjabi language. The data was produced under the Mediamen publishing agency over the last ten years. The corpus contains work of literature including short stories, novels, fiction, non-fiction, and dramas. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: pnb

Task: NLP

Format: TXT

Size: 1.82 MB

Mozilla Common Voice

Common Voice Scripted Speech 27.0 - Punjabi

A collection of read speech recordings in Punjabi.

License: CC0-1.0

Locale: pa-IN

Task: ASR

Format: MP3

Size: 104.81 MB

Chishti Sons

Chishti Sons Punjabi Literature Corpus

This corpus is a collection of more than one million tokens of Western Punjabi language. The data was produced under the Chishti Sons publishing agency. The corpus contains work of literature including short stories, stories, fiction, non-fiction, and other literary books. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: pnb

Task: NLP

Format: TXT

Size: 1.65 MB

MirasAI

Punjabi 10 Hours TTS

10-hour Punjabi TTS dataset in Shahmukhi script with paired audio and transcripts, designed for speech synthesis and Punjabi language technology.

License: CC-BY-NC-SA-4.0

Locale: pnb

Task: TTS

Format: WEBM, TSV

Size: 481.96 MB

Tamahi Suneha Magazine

Punjabi Literature Corpus

This corpus contains 10,39,430 tokens of Punjabi Shahmukhi script.

License: CC-BY-NC-4.0

Locale: pa-PK

Task: OTH

Format: TXT

Size: 1.83 MB

MirasAI

Associated Press Pakistan 21.4 Million Pashto Corpus

Pashto news corpus from the Associated Press of Pakistan, containing 99,495 articles and 21.4 million words from 2015–2026.

License: CC-BY-NC-SA-4.0

Locale: pus

Task: NLP

Format: TXT

Size: 38.81 MB

PROXIMA

Synthetic Urdu 2

Synthetic Urdu sentences audio dataset of 85,327 verse style audio-text pairs.

License: CC-BY-NC-SA-4.0

Locale: urd

Task: TTS

Format: Parquet

Size: 14.80 GB

MDC DatasetsCompensated

🚧 Northern Pashto Multimodal Dataset 🚧

A Northern Pashto multimodal dataset with 10 hours of synchronized video, audio, transcriptions, speech segmentation, and ELAN annotations.

License: MDC-DLA-1.0

Locale: pbu

Task: NLP

Format: MP4, WAV, EAF

Size: 251.41 GB

PROXIMA

PII Redact Urdu UOPF

Curated Urdu dataset of 100,000 records with fabricated PII spans, for training and evaluating privacy span detection across 12 domains.

License: CC-BY-NC-SA-4.0

Locale: urd

Task: NLP

Format: JSONL

Size: 5.77 MB

MDC DatasetsCompensated

Single Speaker Urdu TTS dataset (Female Speaker, South Punjab)

A high-quality single-speaker Urdu text-to-speech (TTS) dataset with 25 hours of paired speech recordings and transcriptions for speech synthesis.

License: MDC-DLA-1.0

Locale: ur

Task: TTS

Format: WAV, TSV

Size: 6.02 GB

MirasAI

Associated Press Pakistan 74.4 Million Urdu Corpus

A 74.4M-word, 267,887-article Urdu news corpus from Associated Press Pakistan (2015-2026), released for non-commercial research with APP permission.

License: CC-BY-NC-SA-4.0

Locale: urd

Task: NLP

Format: TXT

Size: 143.39 MB

Audio Speech Language Processing lab(ASLP NPU)

UrduSpeech

A 156-hour, multi-domain speech corpus with 12 paralinguistic labels for standard Urdu, code-switched Urdu-English, and Pakistani-accented English.

License: CC-BY-NC-4.0

Locale: ur, en

Task: NLP

Format: .wav, .jsonl

Size: 71.87 GB

Mozilla Common Voice

Common Voice Scripted Speech 27.0 - Pashto

A collection of read speech recordings in Pashto.

License: CC0-1.0

Locale: ps

Task: ASR

Format: MP3

Size: 97.70 GB

MDC DatasetsCompensated

🚧 Northern Pashto Corpus for Speech Recognition 🚧

A Northern Pashto ASR dataset with approximately 10 hours of audio with transcriptions.

License: MDC-DLA-1.0

Locale: pbu

Task: ASR

Format: MP3, TSV

Size: 514.50 MB

Rana Printers Multan

Rana Printers Urdu Literature Corpus

This corpus comprises 1.68 million tokens of high-quality Urdu text collected over the past decade through Rana Printers. It includes a diverse range of literary genres such as stories, short stories, novels, fiction, non-fiction, poetry, and historical works. All content is shared with the authors’ approval. The dataset is intended to support linguistic research, Urdu language technology development, and the preservation of literary and cultural heritage.

License: CC-BY-NC-4.0

Locale: ur

Task: OTH

Format: TXT

Size: 3.00 MB

Community

Sindhi Parts Of Speech (SiPOS)

A large-scale Sindhi Part-of-Speech (SiPOS) tagging dataset for developing and evaluating NLP models in a low-resource language.

License: CC-BY-NC-4.0

Locale: snd

Task: NLP

Format: TXT

Size: 874.04 KB

Community

Sindhi Word Segmentation (SdSEG)

A sentence-level annotated dataset for training and evaluating sequence labeling models such as CRF, BiLSTM, and transformer-based architectures.

License: CC-BY-NC-4.0

Locale: snd

Task: NLP

Format: JSON

Size: 1.13 MB

Forum for Language Initiatives

Kohistani Shina Word List

A UTF-8 encoded Kohistani Shina dictionary and word list corpus with lexical entries, meanings, and annotations for linguistic research and NLP use.

License: CC-BY-NC-4.0

Locale: plk

Task: NLP

Format: TXT

Size: 394.05 KB

Mozilla Common Voice

Common Voice Scripted Speech 27.0 - Indus Kohistani

A collection of read speech recordings in Indus Kohistani.

License: CC0-1.0

Locale: mvy

Task: ASR

Format: MP3

Size: 448.96 MB

Mozilla Common Voice

Common Voice Scripted Speech 27.0 - Kohistani Shina

A collection of read speech recordings in Kohistani Shina.

License: CC0-1.0

Locale: plk

Task: ASR

Format: MP3

Size: 330.89 MB

Mozilla Common Voice

Common Voice Spontaneous Speech 5.0 - Pashto

A collection of spontaneous responses to questions in Pashto.

License: CC0-1.0

Locale: ps

Task: ASR

Format: MP3

Size: 772.19 MB

Mozilla Common Voice

Common Voice Scripted Speech 27.0 - Khetrani

A collection of read speech recordings in Khetrani.

License: CC0-1.0

Locale: xhe

Task: ASR

Format: MP3

Size: 196.14 MB

Balochistan Educational and Cultural Organization

NAWA-E-WATAN Balochi Newspaper Corpus

A ~1.02M-token Balochi newspaper corpus from NAWA-E-WATAN, representing contemporary journalistic and public discourse.

License: CC-BY-NC-4.0

Locale: bgn

Task: NLP

Format: TXT

Size: 1.43 MB

All Datasets

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

Mediamen Punjabi Literature Corpus

Common Voice Scripted Speech 27.0 - Punjabi

Chishti Sons Punjabi Literature Corpus

Punjabi 10 Hours TTS

Punjabi Literature Corpus

Weekly dataset digest

Associated Press Pakistan 21.4 Million Pashto Corpus

Synthetic Urdu 2

🚧 Northern Pashto Multimodal Dataset 🚧

PII Redact Urdu UOPF

Single Speaker Urdu TTS dataset (Female Speaker, South Punjab)

Associated Press Pakistan 74.4 Million Urdu Corpus

UrduSpeech

Common Voice Scripted Speech 27.0 - Pashto

🚧 Northern Pashto Corpus for Speech Recognition 🚧

Rana Printers Urdu Literature Corpus

Sindhi Parts Of Speech (SiPOS)

Sindhi Word Segmentation (SdSEG)

Kohistani Shina Word List

Common Voice Scripted Speech 27.0 - Indus Kohistani

Common Voice Scripted Speech 27.0 - Kohistani Shina

Common Voice Spontaneous Speech 5.0 - Pashto

Common Voice Scripted Speech 27.0 - Khetrani

NAWA-E-WATAN Balochi Newspaper Corpus