śmigiel - machine generated text detection

Fields

For fuller model details, refer to models section. Strategy parameters are explained at strategies section.

Splits

The dataset is split between train, test_a and test_b.

Curation Rationale

The dataset was originally compiled for Shared Task 1 at PolEval 2025 to

1. provide training data for SUPERVISED subtask;

2. provide two sets of testing data for benchmarking submissions;

Source Data

The original, human-written texts are coming from 12 distinct datasources. Collectively, they cover wide range of styles, lengths, and purposes of text-writing.

Domains

To help to balance the detaset in terms of linguistic features, the original sources were grouped into 6 genres or "domains". This balancing is two-fold - it's inward, as the origin of texts within a domain is balanced how much was possible, and outward, as the main part of the data, (train and alpha test subgroup) consists of equal share of the 4 domains. The two remaining ones, news articles and parlimentary hearings, were introduced as part of robust training subset.

Models

To provide for versitile MGT, we used models coming from different families, and of varying sizes.

Strategies

We foster versitality of (generated) data by applying different decoding strategies. These strategies condition how "next token candidates" or strings of thereof are ultimately selected by the model. Below we provide their rundown, together with how they translate into parametrs of model inference's call.

Composition

Test A and Test B are both composed out of thee distinct subsets of data - alpha, beta, and gamma. The two tests differ in proportions of sampled data.

• alpha is simply the test split of the base postprocessing run

• beta is a result of processing human texts from the dev split with a single model (llama-lg, Llama 3.3 70B)

• gamma uses all the available models to process data from unseen datasets (the data was not published before)

The subsets contribute to the two resulting test groups through a round-robin assignment method. While the contribution of alpha is equal in both, the larger Test B gets more content from beta and gamma.

FAQ

Who are the source data producers?

The source data comes from open-sourced corpora authors as well as researchers at IPI PAN to whom we are deeply indepted.

Personal and Sensitive Information

The usernames in texts coming from social media have been anonimized. To our best knowledge, the data does not contain any other personal and / or sensitive information.

Data Collection and Processing

The source data is processed to produce prompts to guide machine generation. The generations then further filtrated and sampled from. For details on the processes please refer to our repo.

Citation

@article{
  title={Śmigiel Dataset: Laying Foundations for Investigating Machine-Generated Text Detection in Polish},
  author={Jakub Strebeyko, Alina Wróblewska, Piotr Przybyła},
  journal={LREC 2025},
  year={2025}
}

• Designed by: Piotr Przybyła, PhD and Alina Wróblewska, PhD

• Compiled and Curated by: JJ Strebeyko

• Language(s) (NLP): Polish

• License: Creative Commons Attribution 4.0

• Repository: https://github.com/JStrebeyko/code-of-smigiel

• Paper: PolEval 2025 Task 1 Śmigiel: Spotting Machine-Generated Text from LLMs for Polish

• Resources: Zenodo

Acknowledgements

Dataset creation was the Ramón y Cajal grant RYC2024-050327-I, funded by the Spanish State Research Agency (MI-CIU/AEI/10.13039/501100011033) and by the European Social Fund Plus (ESF+) of the European Union. We also gratefully acknowledge Polish high-performance computing infrastructure PLGrid (HPC Centers: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PLG/2025/018019.