Datasets | Mozilla Data Collective

A diverse Hindi text corpus spanning literary works and news article blogs, curated for NLP research and language modeling tasks.

Hindi Literature & News Corpus

License: CC-BY-NC-SA-4.0

Locale: hin

Task: NLP

Format: TXT

Size: 19.52 MB

Hindi monolingual corpus of about 10 million tokens, compiled from multiple authors and including literary works and personal articles.

Hindi 10 Million Text Corpus

License: CC-BY-ND-4.0

Locale: hin

Task: NLP

Format: DOCX

Size: 21.91 MB

A curated Marathi text corpus containing diverse written content for natural language processing, linguistic research, and language modeling tasks.

Marathi Blog & Literature Corpus

License: CC-BY-NC-4.0

Locale: mar

Task: NLP

Format: TXT

Size: 4.20 MB

Collaborative Action For Research & Development (CARD)

IBT Torwali Wordlist

The IBT Torwali Wordlist contains approximately 20,000 unique entries in Torwali (ISO 639-3: trw), an under-documented Indo-Aryan language spoken in northern Pakistan. The dataset comprises standardized lexical entries covering core vocabulary, function words, and culturally salient terms, with consistent orthography and normalization suitable for linguistic and computational use. Entries are aligned with English and Urdu glosses, and include part-of-speech tag.

License: CC-BY-SA-4.0

Locale: trw

Task: NLP

Format: CSV

Size: 312.87 KB

RFE/RL

RFE/RL Pashto (Pakistani) News Text Corpus

Longitudinal Pakistani Pashto news corpus from Radio Mashaal (2010-2026) with over 52,000 articles and 14.5M tokens.

License: CC-BY-NC-SA-4.0

Locale: ps

Task: NLP

Format: TXT

Size: 39.26 MB

10-hour Punjabi TTS dataset in Shahmukhi script with paired audio and transcripts, designed for speech synthesis and Punjabi language technology.

Punjabi 10 Hours TTS

License: CC-BY-NC-SA-4.0

Locale: pnb

Task: TTS

Format: WEBM, TSV

Size: 481.96 MB

This Mandar language dataset is text and audio data paired for languange documentation and linguistic research in technology.

Mandar Balanipa Speech Corpus

License: CC-BY-NC-4.0

Locale: mdr

Task: TTS

Format: WEBM, TSV

Size: 281.70 MB

A Khowar-English speech translation dataset with approximately 9.5 hours of Khowar audio with English text translations.

🚧 Khowar Speech Translation Corpus 🚧

License: MDC DLA-1.0

Locale: khw, en

Task: MT

Format: MP3, TSV

Size: 244.96 MB

Institute of African Digital Humanities

Bamun-French Parallel Corpus 1.1

This dataset is an updated version of the "Bamun-French Parallel Corpus", a parallel corpus of texts in Bamun (Shupament) and French.

License: NOODL-1.0

Locale: bax

Task: MT

Format: TSV

Size: 99.78 KB

A curated Kannada text corpus containing diverse written content for natural language processing, linguistic research, and language technology development.

Kannada Text Corpus

License: CC-BY-NC-SA-4.0

Locale: kan

Task: NLP

Format: TXT

Size: 2.61 MB

Taruen

Kyrgyz Folklore Text Corpus

A 427k-word Kyrgyz folklore corpus of tales, proverbs, and aphorisms, digitized from 5 Bishkek academic volumes (2016-2017) for NLP tasks.

License: CC0-1.0

Locale: ky

Task: NLP

Format: TXT

Size: 1.28 MB

A high-quality single-speaker Urdu text-to-speech (TTS) dataset with 25 hours of paired speech recordings and transcriptions for speech synthesis.

Single Speaker Urdu TTS dataset (Female Speaker, South Punjab)

License: MDC DLA-1.0

Locale: ur

Task: TTS

Format: WAV, TSV

Size: 6.02 GB

A Northern Pashto multimodal dataset with 10 hours of synchronized video, audio, transcriptions, speech segmentation, and ELAN annotations.

🚧 Northern Pashto Multimodal Dataset 🚧

License: MDC DLA-1.0

Locale: pbu

Task: NLP

Format: MP4, WAV, EAF

Size: 251.41 GB

Taruen

Polish Public Domain 20th Century Literature Text Corpus

A 4.2-million-word corpus of 54 iconic Polish novels, multi-volume epics, and documentary prose pieces from the late 19th and early 20th centuries.

License: CC0-1.0

Locale: pl

Task: NLP

Format: TXT

Size: 10.86 MB

CLEAR Global

TWB Parallel Sentence kits - Hausa (30k)

30,000 English–Hausa parallel sentences from CLEAR Global's Gamayun kits, translated by professional translators from Tatoeba source sentences.

License: CC-BY-4.0

Locale: hau

Task: MT

Format: TSV

Size: 1.68 MB

RFE/RL

RFE/RL Tajik News Text Corpus

Longitudinal Tajik & Russian news corpus from Radio Ozodi (2000-2026) with over 166,000 articles and 20M tokens.

License: CC-BY-NC-SA-4.0

Locale: tg,ru

Task: NLP

Format: TXT

Size: 145.27 MB

Biomedical Urdu Named Entity Recognition dataset for advancing healthcare and clinical NLP research in Urdu.

Biomedical Urdu Named Entity Recognition (BioUNER)

License: CC-BY-NC-4.0

Locale: urd

Task: NLP

Format: CSV

Size: 336.69 KB

Tamazight NLP

AmaWar

Amawal Warayni - ⴰⵎⴰⵡⴰⵍ ⴰⵙⵏⵎⴰⵍⴰⵢ ⵏ ⵉⵏⵓⵎⴰⴽ ⵏ ⵡⴰⵙⵙⴰⵖⵏ ⴷ ⵉⵎⵢⴰⴳⵏ ⵏ ⵜⵎⴰⵣⵉⵖⵜ ⵏ ⴰⵢⵜ ⵡⴰⵔⴰⵢⵏ

License: Apache-2.0

Locale: rif

Task: MT

Format: TSV

Size: 100.28 KB

A Khowar ASR dataset with approximately 10 hours of audio with transcriptions.

🚧 Khowar Corpus for Speech Recognition 🚧

License: MDC DLA-1.0

Locale: khw

Task: ASR

Format: MP3, TSV

Size: 245.00 MB

PROXIMA

Synthetic Urdu 2

Synthetic Urdu sentences audio dataset of 85,327 verse style audio-text pairs.

License: CC-BY-NC-SA-4.0

Locale: urd

Task: TTS

Format: Parquet

Size: 14.80 GB

A Hazargi speech translation dataset with approximately 10 hours of Hazargi audio with English text translations.

🚧 Hazargi-English Speech Translation Corpus 🚧

License: MDC DLA-1.0

Locale: haz, en

Task: MT

Format: MP3, TSV

Size: 511.58 MB