Datasets | Mozilla Data Collective

Common Voice

Common Voice Scripted Speech 27.0 - Paiwan

A collection of read speech recordings in Paiwan.

License: CC0-1.0

Locale: pwn

Task: ASR

Format: MP3

Size: 281.77 MB

MEDIAMEN

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

This parallel sentences corpus containing 30,405 aligned sentence pairs with a total of approximately 0.62 million tokens, curated from the archival materials of Mediamen (Advertising Agency). The sentences were professionally translated from English into Punjabi (Shahmukhi) and are intended to support machine translation, linguistic research, and Punjabi language technology development, particularly for real-world and contemporary language use.

License: CC-BY-NC-4.0

Locale: en-PK, pnb

Task: MT

Format: CSV

Size: 1.08 MB

Common Voice

Common Voice Scripted Speech 27.0 - Taiwanese (Minnan)

A collection of read speech recordings in Taiwanese (Minnan).

License: CC0-1.0

Locale: nan-tw

Task: ASR

Format: MP3

Size: 471.51 MB

MEDIAMEN

Mediamen Punjabi Literature Corpus

This corpus is a collection of one million tokens of Western Punjabi language. The data was produced under the Mediamen publishing agency over the last ten years. The corpus contains work of literature including short stories, novels, fiction, non-fiction, and dramas. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: pnb

Task: NLP

Format: TXT

Size: 1.82 MB

Tamazight NLP

AmaWar

Amawal Warayni - ⴰⵎⴰⵡⴰⵍ ⴰⵙⵏⵎⴰⵍⴰⵢ ⵏ ⵉⵏⵓⵎⴰⴽ ⵏ ⵡⴰⵙⵙⴰⵖⵏ ⴷ ⵉⵎⵢⴰⴳⵏ ⵏ ⵜⵎⴰⵣⵉⵖⵜ ⵏ ⴰⵢⵜ ⵡⴰⵔⴰⵢⵏ

License: Apache-2.0

Locale: rif

Task: MT

Format: TSV

Size: 100.28 KB

Common Voice

Common Voice Scripted Speech 27.0 - Pa'O

A collection of read speech recordings in Pa'O.

License: CC0-1.0

Locale: blk

Task: ASR

Format: MP3

Size: 463.99 KB

Common Voice

Common Voice Scripted Speech 27.0 - Chinese (Taiwan)

A collection of read speech recordings in Chinese (Taiwan).

License: CC0-1.0

Locale: zh-TW

Task: ASR

Format: MP3

Size: 2.96 GB

MirasAI

Associated Press Pakistan 21.4 Million Pashto Corpus

Pashto news corpus from the Associated Press of Pakistan, containing 99,495 articles and 21.4 million words from 2015–2026.

License: CC-BY-NC-SA-4.0

Locale: pus

Task: NLP

Format: TXT

Size: 38.81 MB

Balochistan Educational and Cultural Organization

NAWA-E-WATAN Balochi Newspaper Corpus

A ~1.02M-token Balochi newspaper corpus from NAWA-E-WATAN, representing contemporary journalistic and public discourse.

License: CC-BY-NC-4.0

Locale: bgn

Task: NLP

Format: TXT

Size: 1.43 MB

Chana

Pano: Comparative Word Lists

Twenty-four lexical elicitation recordings from nine Amazonian varieties, providing 7.35 hours of comparative data collected with the same word list.

License: Chana 2.0

Locale: mul

Task: N/A

Format: WAV, MD, CSV

Size: 4.64 GB

OpenCSG

Finweb-Edu-Chinese-v2.2

Fineweb-Edu-Chinese v2.2: Updated Chinese educational web dataset (Fineweb series) — access via www.opencsg.com.

License: Apache-2.0

Locale: zh

Task: LLM

Format: parquet

Size: 624.68 MB

Chana

Awajún: Phonetic Word List

150 Awajún lexical items with WAV recordings and Praat TextGrids, part of a controlled comparative phonetic dataset of three Peruvian languages.

License: Chana 2.0

Locale: agr

Task: N/A

Format: WAV, TextGrid, MD, CSV

Size: 70.83 MB

Common Voice

Common Voice Spontaneous Speech 5.0 - Chinese (China)

A collection of spontaneous responses to questions in Chinese (China).

License: CC0-1.0

Locale: zh-CN

Task: ASR

Format: MP3

Size: 121.91 KB

OpenCSG

chinese-cosmopedia

A large-scale high-quality Chinese text dataset developed by OpenCSG, containing ~15 million entries (≈60B tokens) covering multi-domain content (encyclopedia, education, etc.). Cleaned and deduplicated to remove low-quality content, it is optimized for large language model pretraining, text generation, and other Chinese NLP downstream tasks, compatible with mainstream toolchains (Hugging Face Datasets, PyTorch).

License: Apache-2.0

Locale: zh

Task: LLM

Format: parquet

Size: 6.09 GB

Common Voice

Common Voice Spontaneous Speech 5.0 - Puno Quechua

A collection of spontaneous responses to questions in Puno Quechua.

License: CC0-1.0

Locale: qxp

Task: ASR

Format: MP3

Size: 700.53 MB

MDC DatasetsCompensated

🚧 Southeast Asian Cantonese TTS dataset 🚧

2 hours of studio-quality, deliberate, read speech and corresponding text in Southeast Asian Cantonese, from a single speaker.

License: MDC DLA-1.0

Locale: yue

Task: TTS

Format: WAV, TSV

Size: 202 B

Tamahi Suneha Magazine

Punjabi Literature Corpus

This corpus contains 10,39,430 tokens of Punjabi Shahmukhi script.

License: CC-BY-NC-4.0

Locale: pa-PK

Task: OTH

Format: TXT

Size: 1.83 MB

Common Voice

Common Voice Scripted Speech 27.0 - Puno Quechua

A collection of read speech recordings in Puno Quechua.

License: CC0-1.0

Locale: qxp

Task: ASR

Format: MP3

Size: 689.31 MB

Chishti Sons

Chishti Sons Punjabi Literature Corpus

This corpus is a collection of more than one million tokens of Western Punjabi language. The data was produced under the Chishti Sons publishing agency. The corpus contains work of literature including short stories, stories, fiction, non-fiction, and other literary books. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: pnb

Task: NLP

Format: TXT

Size: 1.65 MB

Common Voice

Common Voice Scripted Speech 27.0 - Cantonese

A collection of read speech recordings in Cantonese.

License: CC0-1.0

Locale: yue

Task: ASR

Format: MP3

Size: 5.99 GB

Chana

Kakataibo: Phonetic Word List

150 Kakataibo lexical items with WAV recordings and Praat TextGrids, including botanically identified items and comparable Aymara and Awajún data.

License: Chana 2.0

Locale: cbr

Task: N/A

Format: WAV, TextGrid, MD, CSV

Size: 65.10 MB

MirasAI

Associated Press Pakistan 1.8 Million Chinese Corpus

A Chinese-language corpus of about 1.8 million Associated Press Pakistan news articles, useful for NLP and language model training.

License: CC-BY-NC-SA-4.0

Locale: zho

Task: NLP

Format: TXT

Size: 2.76 MB

MDC Curators

Common Voice Scripted Speech 26.0 - Beijing Chinese

Chinese Common Voice scripted speech 26.0 segment of Beijing Chinese accents.

License: CC0-1.0

Locale: zh-CN-beijing

Task: ASR

Format: MP3, TSV

Size: 113.39 MB

MDC DatasetsCompensated

🚧 Southeast Asian Hokkien TTS dataset 🚧

2 hours of studio-quality, deliberate, read speech from a single speaker of Southeast Asian Hokkien.

License: MDC DLA-1.0

Locale: nan

Task: TTS

Format: WAV, TSV

Size: 194 B

All Datasets

Not sure where to start?

Common Voice Scripted Speech 27.0 - Paiwan

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

Common Voice Scripted Speech 27.0 - Taiwanese (Minnan)

Mediamen Punjabi Literature Corpus

AmaWar

Common Voice Scripted Speech 27.0 - Pa'O

Common Voice Scripted Speech 27.0 - Chinese (Taiwan)

Associated Press Pakistan 21.4 Million Pashto Corpus

NAWA-E-WATAN Balochi Newspaper Corpus

Pano: Comparative Word Lists

Finweb-Edu-Chinese-v2.2

Awajún: Phonetic Word List

Common Voice Spontaneous Speech 5.0 - Chinese (China)

chinese-cosmopedia

Common Voice Spontaneous Speech 5.0 - Puno Quechua

🚧 Southeast Asian Cantonese TTS dataset 🚧

Punjabi Literature Corpus

Common Voice Scripted Speech 27.0 - Puno Quechua

Chishti Sons Punjabi Literature Corpus

Common Voice Scripted Speech 27.0 - Cantonese

Kakataibo: Phonetic Word List

Associated Press Pakistan 1.8 Million Chinese Corpus

Common Voice Scripted Speech 26.0 - Beijing Chinese

🚧 Southeast Asian Hokkien TTS dataset 🚧