Datasets | Mozilla Data Collective

MDC DatasetsCompensated

🚧 Hazargi-English Speech Translation Corpus 🚧

A Hazargi speech translation dataset with approximately 10 hours of Hazargi audio with English text translations.

License: MDC DLA-1.0

Locale: haz, en

Task: MT

Format: MP3, TSV

Size: 511.58 MB

MDC DatasetsCompensated

🚧 Hazargi-English Parallel Text Corpus 🚧

A Hazargi-English parallel corpus derived from transcribed and translated video recordings

License: MDC DLA-1.0

Locale: haz, en

Task: MT

Format: TSV

Size: 434.04 KB

Keblagh e Azergi

Elkhani Hazargi Literature Corpus

Hazargi literary corpus (~0.5M tokens) of poetry, folklore, and prose texts representing Hazara linguistic and cultural heritage.

License: CC-BY-NC-4.0

Locale: haz

Task: NLP

Format: TXT

Size: 2.46 MB

MDC DatasetsCompensated

🚧 Hazargi Corpus for Speech Recognition 🚧

A Hazargi ASR dataset with approximately 10 hours of audio with transcriptions.

License: MDC DLA-1.0

Locale: haz

Task: ASR

Format: MP3, TSV

Size: 511.56 MB

Habib University

Burushaski-English Speech Translation Corpus

A linguistically informed Burushaski speech - English text corpus containing 14,970 short audio recordings paired with corresponding English text translations.

License: CC-BY-NC-SA-4.0

Locale: bsk

Task: MT

Format: WAV, TXT

Size: 921.47 MB

Habib University

Burushaski-English Speech Translation Corpus

A linguistically informed Burushaski speech - English text corpus containing 14,970 short audio recordings paired with corresponding English text translations.

License: CC-BY-NC-SA-4.0

Locale: bsk

Task: MT

Format: WAV, TXT

Size: 921.47 MB

Keblagh e Azergi

Keblagh-e-Azergi Hazargi literature corpus

This corpus is a collection of more than one hundred thousand tokens of Hazargi language. The corpus contains work of literature, poems, folk and short stories and dramas. The data is being shared with the approval of the authors. It claims to support linguistic research, language technology development and cultural preservation.

License: CC-BY-NC-4.0

Locale: haz

Task: NLP

Format: TXT

Size: 193.28 KB

MDC DatasetsCompensated

🚧 Hazargi Multimodal Dataset 🚧

A Hazargi multimodal dataset with 10 hours of synchronized video, audio, transcriptions, speech segmentation, and ELAN annotations.

License: MDC DLA-1.0

Locale: haz

Task: NLP

Format: MP4, EAF

Size: 44.00 GB

Common Voice

Common Voice Scripted Speech 26.0 - Hazargi

A collection of read speech recordings in Hazargi.

License: CC0-1.0

Locale: haz

Task: ASR

Format: MP3

Size: 175.19 MB

Kaleem Art Press

Saraiki-English Parallel Corpus

English–Saraiki parallel corpus: 51,447 aligned sentence pairs (~0.89M words), translated by Kaleem Art Press for MT and Saraiki NLP research.

License: CC-BY-NC-4.0

Locale: mul

Task: MT

Format: CSV

Size: 1.92 MB

MDC DatasetsCompensated

🚧 Khowar Speech Translation Corpus 🚧

A Khowar-English speech translation dataset with approximately 9.5 hours of Khowar audio with English text translations.

License: MDC DLA-1.0

Locale: khw, en

Task: MT

Format: MP3, TSV

Size: 244.96 MB

The University of Melbourne

Hawrami Kurdish TTS dataset 1.0

This dataset contains high-quality single-speaker audio recordings in Hawrami Kurdish (Hewrami, ISO 639-3:hac), also known as the Gorani language, intended for building Text-to-Speech (TTS) and Automatic Speech recognition (ASR) systems. The dataset comprises 5 hours and 15 minutes of aligned audio and text data. Hawrami is classified as Definitely Endangered by UNESCO.

License: CC-BY-4.0

Locale: hac

Task: TTS

Format: WAV

Size: 706.11 MB

MDC DatasetsCompensated

🚧 Khowar-English Parallel Corpus 🚧

A parallel corpus of over 10,000 sentences of Khowar and English. The sentences are derived from a multimodal corpus consisting of 152 transcribed and translated Khowar language videos.

License: MDC DLA-1.0

Locale: khw, en

Task: MT

Format: TSV

Size: 407.57 KB

TAUSCompensated

🚧 Colloquial Parallel Text (English - Hausa) 🚧

100k native-reviewed colloquial sentences translated from US English into Hausa for fine-tuning conversational MT and AI models.

License: TAUS DUA

Locale: en-US, ha-NG

Task: MT

Format: TSV

Size: 4.49 MB

LocaleNLP

English Hausa Parallel Corpus

An English–Hausa dataset with 5,000 sentence pairs useful for machine translation and basic language processing tasks.

License: CC-BY-NC-4.0

Locale: eng, hau

Task: MT

Format: csv

Size: 164.32 KB

TAUSCompensated

🚧 Colloquial Parallel Text (English - Hausa) 🚧

49k native-reviewed colloquial sentences translated from US English into Hausa for fine-tuning conversational MT and AI models.

License: TAUS DUA

Locale: en-US, ha-NG

Task: MT

Format: TSV

Size: 2.93 MB

MEDIAMEN

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

This parallel sentences corpus containing 30,405 aligned sentence pairs with a total of approximately 0.62 million tokens, curated from the archival materials of Mediamen (Advertising Agency). The sentences were professionally translated from English into Punjabi (Shahmukhi) and are intended to support machine translation, linguistic research, and Punjabi language technology development, particularly for real-world and contemporary language use.

License: CC-BY-NC-4.0

Locale: en-PK, pnb

Task: MT

Format: CSV

Size: 1.08 MB

Common Voice

Common Voice Scripted Speech 26.0 - Saraiki

A collection of read speech recordings in Saraiki.

License: CC0-1.0

Locale: skr

Task: ASR

Format: MP3

Size: 133.49 MB

Common Voice

Common Voice Scripted Speech 26.0 - Gujari

A collection of read speech recordings in Gujari.

License: CC0-1.0

Locale: gju

Task: ASR

Format: MP3

Size: 210.21 MB

MDC Curators

CoVoST 2 Persian - English

The dataset includes audio files in Persian and their translations in English.

License: CC-BY-NC-4.0

Locale: fa, en

Task: MT

Format: MP3, TSV

Size: 1.52 GB

Common Voice

Common Voice Spontaneous Speech 4.0 - Pashto

A collection of spontaneous responses to questions in Pashto.

License: CC0-1.0

Locale: ps

Task: ASR

Format: MP3

Size: 223.81 MB

MDC Community Concierge

VoxForge - Persian

16 minutes (157 utterances) of read speech of ‭فارسی‬ (Persian).

License: GPL-3.0

Locale: fa

Task: ASR

Format: WAV, TSV

Size: 58.20 MB

Common Voice

Common Voice Scripted Speech 26.0 - Pashto

A collection of read speech recordings in Pashto.

License: CC0-1.0

Locale: ps

Task: ASR

Format: MP3

Size: 101.84 GB

Common Voice

Common Voice Scripted Speech 26.0 - Gawarbaiti

A collection of read speech recordings in Gawarbaiti.

License: CC0-1.0

Locale: gwt

Task: ASR

Format: MP3

Size: 249.85 MB

All Datasets

Not sure where to start?

🚧 Hazargi-English Speech Translation Corpus 🚧

🚧 Hazargi-English Parallel Text Corpus 🚧

Elkhani Hazargi Literature Corpus

🚧 Hazargi Corpus for Speech Recognition 🚧

Burushaski-English Speech Translation Corpus

Burushaski-English Speech Translation Corpus

Keblagh-e-Azergi Hazargi literature corpus

🚧 Hazargi Multimodal Dataset 🚧

Common Voice Scripted Speech 26.0 - Hazargi

Saraiki-English Parallel Corpus

🚧 Khowar Speech Translation Corpus 🚧

Hawrami Kurdish TTS dataset 1.0

🚧 Khowar-English Parallel Corpus 🚧

🚧 Colloquial Parallel Text (English - Hausa) 🚧

English Hausa Parallel Corpus

🚧 Colloquial Parallel Text (English - Hausa) 🚧

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

Common Voice Scripted Speech 26.0 - Saraiki

Common Voice Scripted Speech 26.0 - Gujari

CoVoST 2 Persian - English

Common Voice Spontaneous Speech 4.0 - Pashto

VoxForge - Persian

Common Voice Scripted Speech 26.0 - Pashto

Common Voice Scripted Speech 26.0 - Gawarbaiti