Datasets | Mozilla Data Collective

Common Voice

Common Voice Scripted Speech 26.0 - Paiwan

A collection of read speech recordings in Paiwan.

License: CC0-1.0

Locale: pwn

Task: ASR

Format: MP3

Size: 281.76 MB

MEDIAMEN

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

This parallel sentences corpus containing 30,405 aligned sentence pairs with a total of approximately 0.62 million tokens, curated from the archival materials of Mediamen (Advertising Agency). The sentences were professionally translated from English into Punjabi (Shahmukhi) and are intended to support machine translation, linguistic research, and Punjabi language technology development, particularly for real-world and contemporary language use.

License: CC-BY-NC-4.0

Locale: en-PK, pnb

Task: MT

Format: CSV

Size: 1.08 MB

Common Voice

Common Voice Scripted Speech 26.0 - Taiwanese (Minnan)

A collection of read speech recordings in Taiwanese (Minnan).

License: CC0-1.0

Locale: nan-tw

Task: ASR

Format: MP3

Size: 469.67 MB

MEDIAMEN

Mediamen Punjabi Literature Corpus

This corpus is a collection of one million tokens of Western Punjabi language. The data was produced under the Mediamen publishing agency over the last ten years. The corpus contains work of literature including short stories, novels, fiction, non-fiction, and dramas. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: pnb

Task: NLP

Format: TXT

Size: 1.82 MB

Tamazight NLP

AmaWar

Amawal Warayni - ⴰⵎⴰⵡⴰⵍ ⴰⵙⵏⵎⴰⵍⴰⵢ ⵏ ⵉⵏⵓⵎⴰⴽ ⵏ ⵡⴰⵙⵙⴰⵖⵏ ⴷ ⵉⵎⵢⴰⴳⵏ ⵏ ⵜⵎⴰⵣⵉⵖⵜ ⵏ ⴰⵢⵜ ⵡⴰⵔⴰⵢⵏ

License: Apache-2.0

Locale: rif

Task: MT

Format: TSV

Size: 100.28 KB

Common Voice

Common Voice Scripted Speech 26.0 - Chinese (Taiwan)

A collection of read speech recordings in Chinese (Taiwan).

License: CC0-1.0

Locale: zh-TW

Task: ASR

Format: MP3

Size: 2.95 GB

Balochistan Educational and Cultural Organization

NAWA-E-WATAN Balochi Newspaper Corpus

A ~1.02M-token Balochi newspaper corpus from NAWA-E-WATAN, representing contemporary journalistic and public discourse.

License: CC-BY-NC-4.0

Locale: bgn

Task: NLP

Format: TXT

Size: 1.43 MB

OpenCSG

Finweb-Edu-Chinese-v2.2

Fineweb-Edu-Chinese v2.2: Updated Chinese educational web dataset (Fineweb series) — access via www.opencsg.com.

License: Apache-2.0

Locale: zh

Task: LLM

Format: parquet

Size: 624.68 MB

Common Voice

Common Voice Scripted Speech 26.0 - Puno Quechua

A collection of read speech recordings in Puno Quechua.

License: CC0-1.0

Locale: qxp

Task: ASR

Format: MP3

Size: 689.37 MB

OpenCSG

chinese-cosmopedia

A large-scale high-quality Chinese text dataset developed by OpenCSG, containing ~15 million entries (≈60B tokens) covering multi-domain content (encyclopedia, education, etc.). Cleaned and deduplicated to remove low-quality content, it is optimized for large language model pretraining, text generation, and other Chinese NLP downstream tasks, compatible with mainstream toolchains (Hugging Face Datasets, PyTorch).

License: Apache-2.0

Locale: zh

Task: LLM

Format: parquet

Size: 6.09 GB

Common Voice

Common Voice Spontaneous Speech 4.0 - Puno Quechua

A collection of spontaneous responses to questions in Puno Quechua.

License: CC0-1.0

Locale: qxp

Task: ASR

Format: MP3

Size: 700.53 MB

Tamahi Suneha Magazine

Punjabi Literature Corpus

This corpus contains 10,39,430 tokens of Punjabi Shahmukhi script.

License: CC-BY-NC-4.0

Locale: pa-PK

Task: OTH

Format: TXT

Size: 1.83 MB

Common Voice

Common Voice Scripted Speech 26.0 - Cantonese

A collection of read speech recordings in Cantonese.

License: CC0-1.0

Locale: yue

Task: ASR

Format: MP3

Size: 5.99 GB

Chishti Sons

Chishti Sons Punjabi Literature Corpus

This corpus is a collection of more than one million tokens of Western Punjabi language. The data was produced under the Chishti Sons publishing agency. The corpus contains work of literature including short stories, stories, fiction, non-fiction, and other literary books. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development, and cultural preservation.

License: CC-BY-NC-4.0

Locale: pnb

Task: NLP

Format: TXT

Size: 1.65 MB

MDC Curators

Common Voice Scripted Speech 26.0 - Beijing Chinese

Chinese Common Voice scripted speech 26.0 segment of Beijing Chinese accents.

License: CC0-1.0

Locale: zh-CN-beijing

Task: ASR

Format: MP3, TSV

Size: 113.39 MB

Aim Foundation

Aim Foundation Dari Literature Corpus

This corpus is a collection of more than seven hundred thousand tokens of Dari language. The corpus contains work of literature including poems, stories, novels, fictional and non-fictional and different articles. The data is being shared with the approval of the authors. It aims to support linguistic research, language technology development and cultural preservation.

License: CC-BY-NC-4.0

Locale: prs

Task: NLP

Format: TXT

Size: 1.74 MB

Common Voice

Common Voice Scripted Speech 26.0 - Chinese (China)

A collection of read speech recordings in Chinese (China).

License: CC0-1.0

Locale: zh-CN

Task: ASR

Format: MP3

Size: 21.39 GB

Common Voice

Common Voice Scripted Speech 26.0 - Huarijio

A collection of read speech recordings in Huarijio.

License: CC0-1.0

Locale: var

Task: ASR

Format: MP3

Size: 236.19 MB

MDC Curators

Common Voice Scripted Speech 26.0 - Wu Chinese

Chinese Common Voice scripted speech 26.0 segment of Wu Chinese accents.

License: CC0-1.0

Locale: zh-CN-shanghai

Task: ASR

Format: MP3, TSV

Size: 191.22 MB

OpenCSG

smoltalk-chinese

SmolTalk-Chinese: A multi-task Chinese conversational dataset covering 19 typical dialogue task scenarios.

License: Apache-2.0

Locale: zh

Task: LLM

Format: parquet

Size: 879.81 MB

Bahasa Bahana

BAHANA-Betawi TTS

This dataset consists of various texts involving Indonesian and English code-mixing and read by Betawi speakers of West Java Province, Indonesia.

License: CC-BY-SA-4.0

Locale: bew

Task: TTS

Format: WEBM, TSV

Size: 316.13 MB

Common Voice

Common Voice Scripted Speech 26.0 - Punjabi

A collection of read speech recordings in Punjabi.

License: CC0-1.0

Locale: pa-IN

Task: ASR

Format: MP3

Size: 104.82 MB

Open Home Foundation

Chitwan 1.0

Text to speech dataset for Nepali, male speaker, approximately 1 hour of read speech.

License: CC0-1.0

Locale: ne-NE

Task: TTS

Format: WEBM

Size: 61.68 MB

RFE/RL

RFE/RL Afghan Pashto News Text Corpus

This dataset is a longitudinal news corpus for the Pashto language, sourced from Radio Azadi, from 2000 to 2026, it contains over 204,000 articles (16M tokens).

License: CC-BY-NC-SA-4.0

Locale: ps

Task: NLP

Format: TXT

Size: 117.68 MB

All Datasets

Not sure where to start?

Common Voice Scripted Speech 26.0 - Paiwan

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

Common Voice Scripted Speech 26.0 - Taiwanese (Minnan)

Mediamen Punjabi Literature Corpus

AmaWar

Common Voice Scripted Speech 26.0 - Chinese (Taiwan)

NAWA-E-WATAN Balochi Newspaper Corpus

Finweb-Edu-Chinese-v2.2

Common Voice Scripted Speech 26.0 - Puno Quechua

chinese-cosmopedia

Common Voice Spontaneous Speech 4.0 - Puno Quechua

Punjabi Literature Corpus

Common Voice Scripted Speech 26.0 - Cantonese

Chishti Sons Punjabi Literature Corpus

Common Voice Scripted Speech 26.0 - Beijing Chinese

Aim Foundation Dari Literature Corpus

Common Voice Scripted Speech 26.0 - Chinese (China)

Common Voice Scripted Speech 26.0 - Huarijio

Common Voice Scripted Speech 26.0 - Wu Chinese

smoltalk-chinese

BAHANA-Betawi TTS

Common Voice Scripted Speech 26.0 - Punjabi

Chitwan 1.0

RFE/RL Afghan Pashto News Text Corpus