Datasets | Mozilla Data Collective

Common Voice

Common Voice Scripted Speech 27.0 - Sakizaya

A collection of read speech recordings in Sakizaya.

License: CC0-1.0

Locale: szy

Task: ASR

Format: MP3

Size: 280.80 MB

TAUSCompensated

🚧 Colloquial Parallel Text (English - Hausa) 🚧

49k native-reviewed colloquial sentences translated from US English into Hausa for fine-tuning conversational MT and AI models.

License: TAUS DUA

Locale: en-US, ha-NG

Task: MT

Format: TSV

Size: 2.93 MB

TAUSCompensated

🚧 Colloquial Parallel Text (English - Hausa) 🚧

100k native-reviewed colloquial sentences translated from US English into Hausa for fine-tuning conversational MT and AI models.

License: TAUS DUA

Locale: en-US, ha-NG

Task: MT

Format: TSV

Size: 4.49 MB

Tamazight NLP

AmaWar

Amawal Warayni - ⴰⵎⴰⵡⴰⵍ ⴰⵙⵏⵎⴰⵍⴰⵢ ⵏ ⵉⵏⵓⵎⴰⴽ ⵏ ⵡⴰⵙⵙⴰⵖⵏ ⴷ ⵉⵎⵢⴰⴳⵏ ⵏ ⵜⵎⴰⵣⵉⵖⵜ ⵏ ⴰⵢⵜ ⵡⴰⵔⴰⵢⵏ

License: Apache-2.0

Locale: rif

Task: MT

Format: TSV

Size: 100.28 KB

Chana

Kakataibo: Traditional Stories

Six traditional Kakataibo stories recorded between 2007 and 2012, featuring forest animals and tsikiu mano, with transcriptions and Spanish translations.

License: Chana 2.0

Locale: cbr

Task: N/A

Format: PDF, WAV, MPEG, MP3, MD, CSV

Size: 508.47 MB

Kaleem Art Press

Kaleem Art Press Saraiki Literature Corpus

This corpus contains approximately one million tokens of Saraiki text curated over the past ten years by Kaleem Art Press. It features a wide range of literary genres, including stories, short stories, novels, fiction, non-fiction, travelogues, poetry, biographies, and historical writings. All content is shared with full author approval. The dataset is intended to support linguistic research, Saraiki language technology development, and the preservation of cultural and literary heritage.

License: CC-BY-NC-4.0

Locale: skr

Task: OTH

Format: TXT

Size: 1.84 MB

Baloch Publishers Multan

Baloch Publishers Saraiki Literature Corpus

This corpus is a collection of one million tokens of Saraiki language. The data was produced under the Baloch Publishers over the last ten years. The corpus contains work of literature including dictionaries, short stories, novels, fiction, non-fiction, and travelogue. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: skr

Task: NLP

Format: TXT

Size: 2.04 MB

CLEAR Global

TWB Parallel Sentence kits - Hausa (30k)

30,000 English–Hausa parallel sentences from CLEAR Global's Gamayun kits, translated by professional translators from Tatoeba source sentences.

License: CC-BY-4.0

Locale: hau

Task: MT

Format: TSV

Size: 1.68 MB

Taruen

Kazakh Proverbs Text Corpus

A corpus of traditional Kazakh proverbs from the Babalar sözi series (vols 65-69), designed for NLP and LLM cultural competence evaluations.

License: CC0-1.0

Locale: kk

Task: NLP

Format: TXT

Size: 592.14 KB

Chana

Kakataibo: Traditional songs

Nine traditional Kakataibo songs from four musical genres, recorded in Yamino, Peru (2010–2012), with time-aligned transcriptions and translations.

License: Chana 2.0

Locale: cbr

Task: N/A

Format: WAV, EAF, PDF, MD, CSV

Size: 340.66 MB

Common Voice

Common Voice Scripted Speech 27.0 - Wadiyara Koli

A collection of read speech recordings in Wadiyara Koli.

License: CC0-1.0

Locale: kxp

Task: ASR

Format: MP3

Size: 221.17 MB

Digital Divide Data

ddd-kenya-luhya-70hrs-asr

A collection of high-quality, crowdsourced audio recordings and verified transcriptions for the Luhya language, produced by Digital Divide Data.

License: CC-BY-4.0

Locale: luy

Task: ASR

Format: WAV, XLSX, TSV

Size: 13.90 GB

Chana

Kakataibo: Phonetic Word List

150 Kakataibo lexical items with WAV recordings and Praat TextGrids, including botanically identified items and comparable Aymara and Awajún data.

License: Chana 2.0

Locale: cbr

Task: N/A

Format: WAV, TextGrid, MD, CSV

Size: 65.10 MB

Institute of African Digital Humanities

A Guiziga Muturwa Sociocultural Dataset

Sociocultural dataset for the Guiziga Muturwa community of Cameroon: identity, kinship, religion, economy, onomastics and a linguistic component.

License: NOODL-1.0

Locale: giz

Task: OTH

Format: TSV, PDF

Size: 75.51 KB

CLEAR Global

TWB Parallel Sentence kits - Swahili (5k)

5,000 English–Swahili parallel sentences from CLEAR Global's Gamayun mini-kit, translated by professional translators from Tatoeba source sentences.

License: CC-BY-4.0

Locale: swa

Task: MT

Format: TSV

Size: 347.61 KB

Common Voice

Common Voice Spontaneous Speech 5.0 - Sabah Bisaya

A collection of spontaneous responses to questions in Sabah Bisaya.

License: CC0-1.0

Locale: bsy

Task: ASR

Format: MP3

Size: 221.30 MB

Community

JAV-Culture: Gresik–Surabaya Multimodal Corpus

A corpus of annotated photographs of cultural objects described in the Gresik dialect of Javanese language.

License: CC-BY-NC-SA-4.0

Locale: jav

Task: CV

Format: JPEG, WEBM, JSON

Size: 713.73 MB

Chana

Kakataibo: Fieldwork 2006-2007

54 Kakataibo fieldwork sessions recorded in 2006–2007, documenting eight speakers in elicitation, narrative, and conversation, with Transcriber annotations.

License: Chana 2.0

Locale: cbr

Task: N/A

Format: WAV, TRS, XLS, PDF

Size: 1.93 GB

Chana

Kakataibo: Audiovisual Narratives

Video documentation of Kakataibo traditional practices, narratives, and language teaching, collected by Roberto Zariquiey.

License: Chana 2.0

Locale: cbr

Task: N/A

Format: WAV, MPEG, PDF

Size: 3.66 GB

Taruen

Uzbek Proverbs Text Corpus

A corpus of traditional Uzbek proverbs digitized from a 2005 volume, designed for NLP tasks and LLM cultural competence evaluations.

License: CC0-1.0

Locale: uz

Task: NLP

Format: TXT

Size: 142.68 KB

Community

JAV IKRAR - Craft and Cultural Heritage Multimodal Corpus

A collection of annotated images capturing public space activities, traditional handicrafts, historical and cultural heritage exhibits in Java.

License: CC-BY-NC-SA-4.0

Locale: jav

Task: CV

Format: JPEG, WEBM, JSON

Size: 1.13 GB

Institute of African Digital Humanities

A Baka Sociocultural Dataset

Sociocultural dataset for the Baka community of Cameroon: identity, kinship, religion, economy, onomastics and a linguistic component.

License: NOODL-1.0

Locale: bkc

Task: OTH

Format: TSV, PDF

Size: 76.14 KB

Common Voice

Common Voice Scripted Speech 27.0 - Asheninka South Ucayali

A collection of read speech recordings in Asheninka South Ucayali.

License: CC0-1.0

Locale: cpy

Task: ASR

Format: MP3

Size: 200.92 MB

Kaleem Art Press

Saraiki-English Parallel Corpus

English–Saraiki parallel corpus: 51,447 aligned sentence pairs (~0.89M words), translated by Kaleem Art Press for MT and Saraiki NLP research.

License: CC-BY-NC-4.0

Locale: mul

Task: MT

Format: CSV

Size: 1.92 MB

All Datasets

Not sure where to start?

Common Voice Scripted Speech 27.0 - Sakizaya

🚧 Colloquial Parallel Text (English - Hausa) 🚧

🚧 Colloquial Parallel Text (English - Hausa) 🚧

AmaWar

Kakataibo: Traditional Stories

Kaleem Art Press Saraiki Literature Corpus

Baloch Publishers Saraiki Literature Corpus

TWB Parallel Sentence kits - Hausa (30k)

Kazakh Proverbs Text Corpus

Kakataibo: Traditional songs

Common Voice Scripted Speech 27.0 - Wadiyara Koli

ddd-kenya-luhya-70hrs-asr

Kakataibo: Phonetic Word List

A Guiziga Muturwa Sociocultural Dataset

TWB Parallel Sentence kits - Swahili (5k)

Common Voice Spontaneous Speech 5.0 - Sabah Bisaya

JAV-Culture: Gresik–Surabaya Multimodal Corpus

Kakataibo: Fieldwork 2006-2007

Kakataibo: Audiovisual Narratives

Uzbek Proverbs Text Corpus

JAV IKRAR - Craft and Cultural Heritage Multimodal Corpus

A Baka Sociocultural Dataset

Common Voice Scripted Speech 27.0 - Asheninka South Ucayali

Saraiki-English Parallel Corpus