Datasets | Mozilla Data Collective

The dataset covers a parallel corpus of 5,173 lines of literal translation text in Indonesian, Javanese multi-dialect, English, and Arabic.

Multidialect of JAV-IND-ENG-ARB Parallel Corpus

License: CC-BY-NC-SA-4.0

Locale: ind,jav,eng,arb

Task: MT

Format: TSV

Size: 566.75 KB

Digital Divide Data

ddd-kenya-somali-68hrs-asr-part1

A collection of high-quality, crowdsourced audio recordings and verified transcriptions for Somali language, produced by Digital Divide Data

License: CC-BY-4.0

Locale: som

Task: ASR

Format: WAV, TSV

Size: 7.68 GB

A 500K token corpus of Noakhalian linguistic data including drama scripts and cultural texts for professional NLP research and language preservation.

Noakhalian (নোয়াখাইল্লা) Text Corpus

License: CC-BY-NC-4.0

Locale: oak

Task: NLP

Format: TXT, DOCX

Size: 3.61 MB

A rare 613.5 K token Rohingya corpus written in the Myanmar script, featuring cultural articles and folklore for advanced NLP research and preservation.

Rohingya Literature Corpus

License: CC-BY-NC-4.0

Locale: rhg

Task: NLP

Format: TXT, DOCX

Size: 7.01 MB

A Hazargi ASR dataset with approximately 10 hours of audio with transcriptions.

🚧 Hazargi Corpus for Speech Recognition 🚧

License: MDC DLA-1.0

Locale: haz

Task: ASR

Format: MP3, TSV

Size: 511.56 MB

Mozilla Data Collective

Sermon-Malaysian-English

7 minutes of Malaysian-accented English speech

License: CC-BY-NC-4.0

Locale: en-MY

Task: ASR

Format: MP4, TXT, SRT

Size: 6.63 MB

Digital Divide Data

ddd-kenya-somali-68hrs-asr-part3

A collection of high-quality, crowdsourced audio recordings and verified transcriptions for the Somali language, produced by Digital Divide Data.

License: CC-BY-4.0

Locale: som

Task: ASR

Format: WAV, TSV

Size: 1.33 GB

Digital Divide Data

ddd-kenya-somali-68hrs-asr-part2

A collection of high-quality, crowdsourced audio recordings and verified transcriptions for the Somali language, produced by Digital Divide Data.

License: CC-BY-4.0

Locale: som

Task: ASR

Format: WAV, TSV

Size: 8.07 GB

Collaborative Action For Research & Development (CARD)

Gawri (گاؤری) Magazine Corpus

The Gawri (گاؤری) Magazine Corpus is a curated collection of monthly Gawri magazine text (~67,724 tokens) that supports research and language technology.

License: CC-BY-NC-4.0

Locale: gwc

Task: NLP

Format: TXT

Size: 146.71 KB

Balochistan Educational and Cultural Organization

Western Balochi Literature Cropus

A cleaned literary corpus in Western Balochi (Rakhshani) including articles, research, translations, and creative literature. (~1.1 m tokens)

License: CC-BY-NC-4.0

Locale: bgn

Task: NLP

Format: TXT

Size: 2.26 MB

Collaborative Action For Research & Development (CARD)

IBT Torwali Literature Corpus

A Torwali literature corpus (~233K tokens) covering poetry, folklore, biographies, and cultural texts for linguistic research and NLP development.

License: CC-BY-NC-4.0

Locale: trw

Task: NLP

Format: TXT

Size: 488.12 KB

This dataset comprises a parallel corpus of 5,173 lines of literal translation text in Indonesian, the Gresik dialect of Javanese, English, and Arabic.

Gresik Dialect of Javanese-IND-ENG-ARB Parallel Corpus

License: CC-BY-NC-SA-4.0

Locale: ind,jav,eng,arb

Task: ML

Format: TSV

Size: 550.10 KB

The languages represented in the corpus include Javanese (Pemalang dialect), Indonesian, English, and Arabic.

Pemalang Dialect of JAV-IND-ENG-ARB Parallel Corpus

License: CC-BY-SA-4.0

Locale: ind,jav,eng,arb

Task: MT

Format: TSV

Size: 555.62 KB

A Sindhi–English parallel corpus in JSONL format, designed for machine translation and cross-lingual NLP tasks across diverse real-world domains.

Sindhi to English Parallel Corpus

License: CC-BY-NC-4.0

Locale: snd, eng

Task: MT

Format: JSONL

Size: 2.50 MB

A Khowar ASR dataset with approximately 10 hours of audio with transcriptions.

🚧 Khowar Corpus for Speech Recognition 🚧

License: MDC DLA-1.0

Locale: khw

Task: ASR

Format: MP3, TSV

Size: 245.00 MB

A 5-hour Kannada speech dataset with time-aligned transcriptions, designed for ASR, forced alignment, and speech research.

Kannada Time Aligned Speech Corpus

License: CC-BY-NC-SA-4.0

Locale: kan

Task: ASR

Format: OGG, SRT

Size: 355.77 MB

This Mandar language dataset is text and audio data paired for languange documentation and linguistic research in technology.

Mandar Balanipa Speech Corpus

License: CC-BY-NC-4.0

Locale: mdr

Task: TTS

Format: WEBM, TSV

Size: 281.70 MB

A Hazargi speech translation dataset with approximately 10 hours of Hazargi audio with English text translations.

🚧 Hazargi-English Speech Translation Corpus 🚧

License: MDC DLA-1.0

Locale: haz, en

Task: MT

Format: MP3, TSV

Size: 511.58 MB

A 500K token corpus of Rangpuri literature and drama provided in UTF-8 format for linguistic research and NLP development.

Rangpuri (অংপুরি Ôṅgpuri) Text Corpus

License: CC-BY-NC-4.0

Locale: rkt

Task: NLP

Format: TXT

Size: 3.68 MB

A corpus of annotated photographs of public spaces, signs, and signage described in Indonesian language.

IND-Sign: Gresik–Surabaya Public Space Multimodal Corpus

License: CC-BY-NC-SA-4.0

Locale: ind

Task: CV

Format: JPEG, WEBM, JSON

Size: 1.55 GB

Hindi monolingual corpus of about 10 million tokens, compiled from multiple authors and including literary works and personal articles.

Hindi 10 Million Text Corpus

License: CC-BY-ND-4.0

Locale: hin

Task: NLP

Format: DOCX

Size: 21.91 MB