Datasets | Mozilla Data Collective

A corpus of annotated photographs of cultural objects described in the Gresik dialect of Javanese language.

JAV-Culture: Gresik–Surabaya Multimodal Corpus

License: CC-BY-NC-SA-4.0

Locale: jav

Task: CV

Format: JPEG, WEBM, JSON

Size: 713.73 MB

Longitudinal Kazakh news corpus from Radio Azattyq (2003-2026) with over 139,000 articles and 35M tokens.

RFE/RL Kazakh News Text Corpus

License: CC-BY-NC-SA-4.0

Locale: kk

Task: NLP

Format: TXT

Size: 126.81 MB

Protham Alo

Prothom Alo Bengali Newspaper Corpus

A 10+ million token Bengali newspaper corpus from Prothom Alo provided in .docx format for large-scale NLP research and linguistic analysis.

License: CC-BY-NC-4.0

Locale: ben

Task: NLP

Format: DOCX

Size: 42.36 MB

A Balti literature corpus preserving literary, cultural, and linguistic heritage for language documentation, research and low-resource NLP applications.

Balti Literature corpus

License: CC-BY-NC-SA-4.0

Locale: bft

Task: NLP

Format: TXT, DOCX

Size: 292.67 KB

Forum for Language Initiatives

Kohistani Shina Word List

A UTF-8 encoded Kohistani Shina dictionary and word list corpus with lexical entries, meanings, and annotations for linguistic research and NLP use.

License: CC-BY-NC-4.0

Locale: plk

Task: NLP

Format: TXT

Size: 394.05 KB

A Sindhi–English parallel corpus in JSONL format, designed for machine translation and cross-lingual NLP tasks across diverse real-world domains.

Sindhi to English Parallel Corpus

License: CC-BY-NC-4.0

Locale: snd, eng

Task: MT

Format: JSONL

Size: 2.50 MB

A diverse Hindi text corpus spanning literary works and news article blogs, curated for NLP research and language modeling tasks.

Hindi Literature & News Corpus

License: CC-BY-NC-SA-4.0

Locale: hin

Task: NLP

Format: TXT

Size: 19.52 MB

CLEAR Global

TWB Parallel Sentence kits - Hausa (30k)

30,000 English–Hausa parallel sentences from CLEAR Global's Gamayun kits, translated by professional translators from Tatoeba source sentences.

License: CC-BY-4.0

Locale: hau

Task: MT

Format: TSV

Size: 1.68 MB

Taruen

Finnish Public Domain 20th Century Literature Text Corpus

A 69.1M-word early 20th-century literature corpus from Project Lönnrot. Predominantly Finnish, with a supplementary Swedish collection.

License: CC0-1.0

Locale: fi, sv

Task: NLP

Format: TXT

Size: 205.76 MB

Balochistan Educational and Cultural Organization

Brahui Research Work Corpus

This corpus consists of research works, academic theses, and scholarly papers, comprising approximately 185,000 tokens. It covers a range of academic topics and formal registers, reflecting standardized writing practices and disciplinary conventions. The corpus is intended to support linguistic analysis, discourse studies, and computational research on academic and research-oriented text.

License: CC-BY-NC-SA-4.0

Locale: brh

Task: NLP

Format: TXT

Size: 1.13 MB

Gawarbati-language literature corpus featuring Diwan Abu Fazl, a poetry collection for documentation and NLP research.

Gawar-Bati Literature Corpus

License: CC-BY-NC-SA-4.0

Locale: gwt

Task: NLP

Format: TXT, DOCX

Size: 876.83 KB

Longitudinal Crimean Tatar news corpus from Qırım.Aqiqat (2014-2026) with over 32,000 articles and 7.5M tokens.

RFE/RL Crimean Tatar News Text Corpus

License: CC-BY-NC-SA-4.0

Locale: crh

Task: NLP

Format: TXT

Size: 18.35 MB

PROXIMA

Synthetic Urdu 2

Synthetic Urdu sentences audio dataset of 85,327 verse style audio-text pairs.

License: CC-BY-NC-SA-4.0

Locale: urd

Task: TTS

Format: Parquet

Size: 14.80 GB

A large-scale Sindhi Part-of-Speech (SiPOS) tagging dataset for developing and evaluating NLP models in a low-resource language.

Sindhi Parts Of Speech (SiPOS)

License: CC-BY-NC-4.0

Locale: snd

Task: NLP

Format: TXT

Size: 874.04 KB

Habib University

Burushaski-English Speech Translation Corpus

A linguistically informed Burushaski speech - English text corpus containing 14,970 short audio recordings paired with corresponding English text translations.

License: CC-BY-NC-SA-4.0

Locale: bsk

Task: MT

Format: WAV, TXT

Size: 921.47 MB

A curated Kannada text corpus containing diverse written content for natural language processing, linguistic research, and language technology development.

Kannada Text Corpus

License: CC-BY-NC-SA-4.0

Locale: kan

Task: NLP

Format: TXT

Size: 2.61 MB

This dataset is a longitudinal news corpus for the Persian language sourced from Radio Farda from 2001 to 2026. It contains over 350,000 articles (51M tokens).

RFE/RL Persian News Text Corpus

License: CC-BY-NC-SA-4.0

Locale: fa

Task: NLP

Format: TXT

Size: 307.78 MB

Longitudinal Azerbaijani & Russian news corpus from Radio Azadlıq (2005-2026) with over 239,000 articles and 37M tokens.

RFE/RL Azerbaijani News Text Corpus

License: CC-BY-NC-SA-4.0

Locale: az,ru

Task: NLP

Format: TXT

Size: 211.65 MB

Habib University

Burushaski-English Speech Translation Corpus

A linguistically informed Burushaski speech - English text corpus containing 14,970 short audio recordings paired with corresponding English text translations.

License: CC-BY-NC-SA-4.0

Locale: bsk

Task: MT

Format: WAV, TXT

Size: 921.47 MB

CLEAR Global

TWB Parallel Sentence kits - Swahili (5k)

5,000 English–Swahili parallel sentences from CLEAR Global's Gamayun mini-kit, translated by professional translators from Tatoeba source sentences.

License: CC-BY-4.0

Locale: swa

Task: MT

Format: TSV

Size: 347.61 KB

Digital Divide Data

ddd-kenya-somali-68hrs-asr-part1

A collection of high-quality, crowdsourced audio recordings and verified transcriptions for Somali language, produced by Digital Divide Data

License: CC-BY-4.0

Locale: som

Task: ASR

Format: WAV, TSV

Size: 7.68 GB

CLEAR Global

TWB Parallel Sentence kits - Kanuri (5k)

5,000 English–Kanuri parallel sentences from CLEAR Global's Gamayun mini-kit, translated by professional translators from Tatoeba source sentences.

License: CC-BY-4.0

Locale: kau

Task: MT

Format: TSV

Size: 358.46 KB

A curated Marathi text corpus containing diverse written content for natural language processing, linguistic research, and language modeling tasks.

Marathi Blog & Literature Corpus

License: CC-BY-NC-4.0

Locale: mar

Task: NLP

Format: TXT

Size: 4.20 MB