wordfreq Spanish word-frequency list — as used in Lost in Transcription (Spanish–Nahuatl)

wordfreq Spanish word-frequency list — as used in Lost in Transcription (Spanish–Nahuatl track)

This archive re-hosts the Spanish word-frequency data of the wordfreq library (version 3.1.1), exported to plain TSV, together with the exact word list derived from it that was used by participant inyongkhafid in the Lost in Transcription challenge (Mozilla Data Collective / DrivenData, 2026), Spanish–Nahuatl track. The competition requires external data that is not already on MDC to be published on MDC; this upload fulfils that rule. The data was not used for training: it served only as a lexicon in a text post-processing step at inference time (see "How the data was used").

Original resource: wordfreq by Robyn Speer, https://github.com/rspeer/wordfreq (also on PyPI). Code: Apache-2.0. Data: Creative Commons Attribution-ShareAlike 4.0 (CC BY-SA 4.0). The data is a snapshot of word usage up to about 2021; the author states that it is unlikely to be updated again.

Contents

All TSV files are UTF-8, tab-separated, with a header row. Columns of the two full lists: word, zipf_frequency (base-10 logarithm of the frequency per billion words, as returned by wordfreq.zipf_frequency), frequency_per_million (as returned by wordfreq.word_frequency × 10⁶). The frequencies are the values wordfreq stores for each word (bucketed on a logarithmic scale), so they reproduce the library's output exactly. Columns of the derived list: word, zipf_frequency.

How the data was used

• The word list was built once with wordfreq 3.1.1: [w for w in top_n_list('es', 80000) if zipf_frequency(w, 'es') >= 3.5] → 16,740 words (equivalently: all words of wordfreq_es_large.tsv with zipf_frequency ≥ 3.5).
• Merged with the Spanish words of the Tetelancingo Nahuatl corpus (already on MDC), it formed a Spanish lexicon for the post-processing step of the submission: transcripts of the additional Nahuatl corpora use a different spelling convention than the competition transcripts, so Nahuatl words were converted to the competition spelling, while any token found in the Spanish lexicon was left unchanged (Spanish words keep their Spanish spelling). The same lexicon was used to protect Spanish stems of at least five letters inside mixed words.
• Nothing in this archive was used to train or fine-tune a model.

Sources of the frequency data (as documented by the wordfreq project)

Wikipedia; subtitles (OpenSubtitles 2018, SUBTLEX); news (NewsCrawl 2014, GlobalVoices); books (Google Books Ngrams 2012); web text (OSCAR); Twitter; Reddit. The wordfreq README asks that Google Books Ngram Viewer, OpenSubtitles and the SUBTLEX authors be credited, and that it remain clear that SUBTLEX is freely available data.

Licence and attribution

Released under CC BY-SA 4.0, the licence of the original data. Changes made: the data is exported from wordfreq's internal format to TSV (same words, same frequencies), one derived file (wordfreq_es_lexicon_as_used.tsv) is added, and this README. No other terms are added: CC BY-SA 4.0 does not permit additional restrictions.

Please cite wordfreq:

Robyn Speer. (2022). rspeer/wordfreq: v3.0 (v3.0.2). Zenodo. https://doi.org/10.5281/zenodo.7199437

Legal basis

Original source: https://github.com/rspeer/wordfreq (also distributed on PyPI). Non-exclusive. The wordfreq data is released by its author under CC BY-SA 4.0. Section 2(a)(1) of this licence lets anyone reproduce and share the material, in whole or in part, including for commercial purposes, on the conditions of attribution (Section 3(a)) and ShareAlike (Section 3(b)). This upload meets both: it keeps the same licence, credits the author, the project and the underlying sources, and reproduces the Spanish word lists in full.