Datasets | Mozilla Data Collective

Aim Foundation

Dari Literature Corpus by Anjuman e Adabi Nayestan

A ~1 M-token Dari (Afghan Persian) literary corpus compiled by Anjuman e Adabi Nayestan, covering prose, poetry, and cultural texts in Perso-Arabic script.

License: CC-BY-NC-4.0

Locale: prs

Task: NLP

Format: TXT

Size: 12.67 MB

Kaleem Art Press

Saraiki-English Parallel Corpus

English–Saraiki parallel corpus: 51,447 aligned sentence pairs (~0.89M words), translated by Kaleem Art Press for MT and Saraiki NLP research.

License: CC-BY-NC-4.0

Locale: mul

Task: MT

Format: CSV

Size: 1.92 MB

Aim Foundation

Aim Foundation Dari Literature Corpus

This corpus is a collection of more than seven hundred thousand tokens of Dari language. The corpus contains work of literature including poems, stories, novels, fictional and non-fictional and different articles. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development and cultural preservation.

License: CC-BY-NC-4.0

Locale: prs

Task: NLP

Format: TXT

Size: 1.74 MB

Community

Hindi 10 Million Text Corpus

Hindi monolingual corpus of about 10 million tokens, compiled from multiple authors and including literary works and personal articles.

License: CC-BY-ND-4.0

Locale: hin

Task: NLP

Format: DOCX

Size: 21.91 MB

Community

Podcast Hari Minggoean (Indonesia)

This dataset is derived from the "Hari Minggoean" podcast, featuring over ten hours of recorded speech from a single, consistent speaker. The content, tailored for a young Indonesian audience, is presented in Indonesian (Bahasa Indonesia) characterized by code-switching with English and a discernible Javanese accent. The collection is comprised of 42 individual audio files (10+ hours). Sample Tapi dari pelafalan, dari intonasi, dari jedanya dia bicara. That's really good. Dan aku sebenarnya suka banget ketika dia ngomong. Yang btw, soal ekspresif tadi. Aku jadi kepikiran deh.

License: CC-BY-SA-4.0

Locale: id-ID

Task: ASR

Format: mp3

Size: 338.92 MB

Community

Ngapak Dialect of JAV-IND-ENG-ARB Parallel Corpus

This dataset consists of a parallel corpus of Javanese translations of the Ngapak dialect from the city of Tegal in Central Java, Indonesia.

License: CC-BY-SA-4.0

Locale: ind,jav,eng,arb

Task: MT

Format: TSV

Size: 544.29 KB

Community

IndoLearner-English Speech Corpus

A corpus of read speech by learners of English language from Indonesia.

License: CC-BY-SA-4.0

Locale: en

Task: CALL

Format: WAV, WEBM, TSV

Size: 1.66 GB

Community

Indonesian-Learner-English

A corpus of English read speech by Indonesian learners.

License: CC-BY-NC-SA-4.0

Locale: en

Task: CALL

Format: WAV, WEBM, TSV

Size: 1.12 GB

Institute of Finno-Ugric/Uralic Studies, University of Hamburg

INEL Nganasan Speech Corpus

A 38.5-hour Speech-to-Text dataset for Nganasan, an endangered Samoyedic language. Features audio aligned with text transcriptions.

License: CC-BY-NC-SA-4.0

Locale: nio

Task: ASR

Format: TSV, MP3

Size: 1.29 GB

Balochi Academy

Balochi Academy Text Corpus

This corpus contains approximately 500k tokens of text from novels, poetry, articles, riddles, and proverbs, covering both literary and traditional genres. It is intended for linguistic research, NLP tasks (e.g., language modeling and text analysis), and cultural documentation.

License: CC-BY-NC-SA-4.0

Locale: bgn

Task: NLP

Format: TXT

Size: 1.88 MB

MirasAI

Sylheti Text Corpus by Haque Publishers

A 524K token Sylheti corpus featuring drama scripts and cultural texts in UTF-8 format for linguistic research and NLP development.

License: CC-BY-NC-4.0

Locale: syl

Task: NLP

Format: TXT

Size: 3.53 MB

Community

Mandar Spontaneous Speech

This dataset is a compilation of spontaneous Mandar speech featuring Indonesian code-switching.

License: CC-BY-NC-4.0

Locale: mdr

Task: ASR

Format: MP3, TSV

Size: 534.45 MB

Community

Multidialect of JAV-IND-ENG-ARB Parallel Corpus

The dataset covers a parallel corpus of 5,173 lines of literal translation text in Indonesian, Javanese multi-dialect, English, and Arabic.

License: CC-BY-NC-SA-4.0

Locale: ind,jav,eng,arb

Task: MT

Format: TSV

Size: 566.75 KB

Community

Demakan Dialect of JAV-IND-ENG-ARB Parallel Corpus

The Demakan dialect is a regional dialect of the Javanese language spoken by the people of the Demak regency in Central Java, Indonesia

License: CC-BY-SA-4.0

Locale: ind,jav,eng,arb

Task: MT

Format: TSV

Size: 555.42 KB

Bahasa Bahana

BAHANA-Betawi TTS

This dataset consists of various texts involving Indonesian and English code-mixing and read by Betawi speakers of West Java Province, Indonesia.

License: CC-BY-SA-4.0

Locale: bew

Task: TTS

Format: WEBM, TSV

Size: 316.13 MB

CLEAR Global

Kanuri Books Corpus

Randomized sentences from Kanuri-language books by four authors, containing 10,281 sentences and 90,706 words.

License: CC-BY-4.0

Locale: kr

Task: LM

Format: TXT

Size: 545.68 KB

MDC Datasets

Lost In Transcription Competition - Development Data (Javanese-Indonesian)

Javanese-Indonesian development dataset for the "Lost In Transcription" competition.

License: CC-BY-SA-4.0

Locale: jv, id

Task: ASR

Format: MP3, TSV

Size: 69.40 MB

MDC DatasetsCompensated

🚧 Khowar-English Parallel Corpus 🚧

A parallel corpus of over 10,000 sentences of Khowar and English. The sentences are derived from a multimodal corpus consisting of 152 transcribed and translated Khowar language videos.

License: MDC DLA-1.0

Locale: khw, en

Task: MT

Format: TSV

Size: 407.57 KB

EELLAK - GreekFOSS

Greek PhD Theses Corpus v1.0

The Greek PhD Theses Corpus is a large, AI-ready dataset

License: CC-BY-NC-SA-4.0

Locale: gr-GR

Task: NLP

Format: JASONL

Size: 7.02 GB

Kaleem Art Press

Multilingual Religious Parallel Corpus (Kaleem Art Press)

This dataset is a multilingual parallel sentences corpus containing 6,465 aligned sentence units with approximately 0.98 million words, curated from Kaleem Art Press archives. It includes parallel religious text data in Arabic, Urdu, Saraiki (standard and dialectal), Punjabi (Shahmukhi), and English, supporting research in machine translation, comparative linguistics, digital humanities, and low-resource language studies.

License: CC-BY-SA-4.0

Locale: mul

Task: MT

Format: CSV

Size: 2.27 MB

MDC DatasetsCompensated

🚧 Hazargi-English Parallel Text Corpus 🚧

A Hazargi-English parallel corpus derived from transcribed and translated video recordings

License: MDC DLA-1.0

Locale: haz, en

Task: MT

Format: TSV

Size: 434.04 KB

Baloch Publishers Multan

Baloch Publishers Saraiki Literature Corpus

This corpus is a collection of one million tokens of Saraiki language. The data was produced under the Baloch Publishers over the last ten years. The corpus contains work of literature including dictionaries, short stories, novels, fiction, non-fiction, and travelogue. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: skr

Task: NLP

Format: TXT

Size: 2.04 MB

MDC Curators

CoVoST 2 English - Indonesian

The dataset includes audio files in English and their translations in Indonesian.

License: CC-BY-NC-4.0

Locale: en, id

Task: MT

Format: MP3, TSV

Size: 12.56 GB

Balochistan Educational and Cultural Organization

Talar (تلار) Barahui Magazine Corpus

A ~150,000-word Brahui corpus from the monthly magazine Talar, covering editorials, essays, fiction, poetry, and socio-cultural commentary.

License: CC-BY-NC-SA-4.0

Locale: brh

Task: NLP

Format: TXT

Size: 317.22 KB

All Datasets

Not sure where to start?

Dari Literature Corpus by Anjuman e Adabi Nayestan

Saraiki-English Parallel Corpus

Aim Foundation Dari Literature Corpus

Hindi 10 Million Text Corpus

Podcast Hari Minggoean (Indonesia)

Ngapak Dialect of JAV-IND-ENG-ARB Parallel Corpus

IndoLearner-English Speech Corpus

Indonesian-Learner-English

INEL Nganasan Speech Corpus

Balochi Academy Text Corpus

Sylheti Text Corpus by Haque Publishers

Mandar Spontaneous Speech

Multidialect of JAV-IND-ENG-ARB Parallel Corpus

Demakan Dialect of JAV-IND-ENG-ARB Parallel Corpus

BAHANA-Betawi TTS

Kanuri Books Corpus

Lost In Transcription Competition - Development Data (Javanese-Indonesian)

🚧 Khowar-English Parallel Corpus 🚧

Greek PhD Theses Corpus v1.0

Multilingual Religious Parallel Corpus (Kaleem Art Press)

🚧 Hazargi-English Parallel Text Corpus 🚧

Baloch Publishers Saraiki Literature Corpus

CoVoST 2 English - Indonesian

Talar (تلار) Barahui Magazine Corpus