StackOverflow Vector Database - rust

Stack2Graph vector database: rust

This release contains 49786 text chunks from Stack Overflow with dense and sparse embeddings, selected by the producer's programming-language memberships. Memberships can overlap: a chunk may occur in several language datasets. These labels describe programming languages, not the natural language of the text. extra-languages, when present, contains source rows without a language match. An absent language is never replaced with an empty dataset.

Files and fields

data/vectors-*.parquet contains Zstandard-compressed tables. Each row has a stable id, question_id, chunk_index, embedding_text, model token counts, languages, tags, dense, sparse_indices, sparse_values and payload_json. The last field preserves the complete original Qdrant payload, including source snapshot identity, source dates, scores and answer flags when available. Dense values are float32 representations of the producer's stored float16 values; this does not recover precision discarded during embedding. Sparse weights preserve float32 values and uint32 token indices.

On Hugging Face, load the vectors config with datasets.load_dataset and use streaming for large releases. The dataset viewer uses only the explicitly listed Parquet files. data/dataset.tar.gz is the portable package also used by MDC: extract it to obtain the same Parquet bytes, README and publication.json. The archive is an exchange dataset, not a Qdrant binary snapshot. Import in bounded batches using each row's id, named dense/sparse vectors and decoded payload_json. Recreate collection settings deliberately for your Qdrant version.

Provenance and reproducibility

• after_id: -9223372036854775808
• chunk_overlap: 128
• chunk_tokens: 1024
• collection: "stackoverflow_questions"
• dense_dimension: 4096
• dense_model: "Qwen/Qwen3-Embedding-8B"
• dense_revision: "1d8ad4ca9b3dd8059ad90a75d4983776a23d44af"
• question_limit: null
• registry_digest: "d2a7357c04e7f47d9bb598189aeeb7db0680d6b35f0d9695d8d2e544121afb8e"
• snapshot_digest: "56fe643952cec74b17ea13d91edc8ad95181326156177c94cce48d3fed9834b9"
• snapshot_id: "stackexchange_20260630_sakura"
• sparse_model: "BAAI/bge-m3"
• :

Export identity: bc9f696c9588560e3be547879c58b96c50c64e3828495471bee43e504392ec7f. Point-ID configuration digest: d16d58c9e7b8007664e3f77b4e9f4a1689a8b8de2c6727b479595621b25e3460. Point IDs use the same snapshot/configuration/question/chunk UUID derivation as the project's Qdrant loader. publication.json lists exact file sizes, SHA-256 hashes, row count, source identity and model/chunking settings. On HF, release.json additionally verifies all managed remote files. Only locally finalized runs with matching stage/report counts are published. The uploader does not re-embed text or query a running database.

Uses and limitations

Intended for dense, sparse and hybrid retrieval, retrieval-augmented generation and reproducible search experiments. Retrieval quality depends on the pinned models and chunking configuration. Overlapping chunks and language memberships are not independent samples; split evaluations by source question to avoid leakage. Source questions can contain personal data, obsolete code, errors and biases. Source-window limits are recorded above. No retrieval benchmark or completeness beyond the recorded source window is claimed.

Attribution and licensing

Derived from Stack Overflow content. Preserve the original post/author attribution and applicable Stack Overflow licensing when redistributing or displaying source content. Question IDs and source snapshot are retained to resolve source posts. This publication declares CC BY-SA 4.0; embedding model licenses and any upstream obligations still apply. Pinned model names and revisions above identify their model cards for review.

Source and procedures: https://github.com/tha-atlas/Stack2Graph. When citing this dataset, include the dataset repository, immutable release revision, source snapshot, export identity, model revisions and the project.

Project citation: Kleybolte, Ventura and Zarcone (2026), Stack2Graph, KG-LLM at LREC26. https://doi.org/10.63317/583q4d3mwbdu.