LLM-Oasis: a large-scale dataset and benchmark for end-to-end factuality evaluation of LLM text (Computational Linguistics 2026)

LLM-Oasis is a resource for training and testing systems that decide whether a whole piece of text is factual, not just a single isolated claim. It is built from Wikipedia: each passage gets a paraphrased factual version and an unfactual version that differs by one subtle falsified claim, giving 81k ⟨factual, unfactual⟩ pairs for training and a human-validated gold benchmark for testing. The task is hard: GPT-4o reaches 60.8% balanced accuracy zero-shot and 68.0% with retrieved Wikipedia evidence, where 50% is chance.

Use it

The public datasets are on the Hugging Face Hub under Babelscape/. Column names and split sizes below come from the Hub’s dataset viewer API.

Train plus validation adds up to about 81K rows per step, in line with the 81,275 pages in the paper. The falsification release is smaller (52,390 rows) than the 81,275 falsified claims reported in Table 2.

The unfactual text dataset already holds both sides of each pair: paraphrase is the factual text and unfactual_text is the version with the falsified claim. That is enough to build a binary training set for an end-to-end factuality classifier:

from datasets import load_dataset

pairs = load_dataset("Babelscape/LLM-Oasis_unfactual_text_generation", split="train")

def to_examples(batch):
    texts = batch["paraphrase"] + batch["unfactual_text"]
    labels = [True] * len(batch["paraphrase"]) + [False] * len(batch["unfactual_text"])
    return {"text": texts, "factual": labels}

train = pairs.map(to_examples, batched=True, remove_columns=pairs.column_names)

# Task 2 gold set: one claim and one evidence passage per row, labels withheld
gold = load_dataset("Babelscape/LLM-Oasis_claim_verification", split="gold")

The gold data for both tasks is released without labels for blind evaluation. For Task 1 the README links a submission form: you upload a .jsonl file with one {"id": ..., "factual": true/false} entry per text, where id matches Babelscape/LLM-Oasis_e2e_factuality_evaluation. At the time of writing, the Hub API reports that dataset as not accessible without authentication.

How it was built

The source is English Wikipedia, restricted to the most visited pages. Each page is split into passages of K = 5 sentences with a sliding window of stride 1, and one passage per page is sampled. GPT-4 then runs four steps on each passage in a single unified prompt (Table 1 of the paper):

The paper’s running example shows the style of error. From a passage on the Amazon rainforest, the claim “Brazil contains 60% of the rainforest” becomes “The majority of the forest is contained within Peru”: another country the rainforest covers, so the error is plausible in context.

The result (Table 2): 81,275 pages, 681,201 claims (8.4 per passage on average), 81,275 falsified claims, and 81,275 factual plus 81,275 unfactual texts of about 83 and 87 tokens.

Human validation. Five expert linguists checked 1,750 instances per step, with 100 shared by all annotators to measure agreement (Table 3):

Gold benchmark. Only instances that no annotator flagged as an error at any stage are kept; texts flagged for omitting a detail are discarded too. From these the paper defines two tasks:

• Task 1, end-to-end factuality evaluation: given a text, output True if it is factual and False if it contains an inaccuracy. Inputs are only the generated factual and unfactual texts, never the original Wikipedia passage. About 2k instances.
• Task 2, evidence-based claim verification: given one claim and one evidence passage (the factual text), output whether the evidence supports the claim. Original claims are labelled True, their falsified versions False.

A baseline built from the training data. The paper also trains a small modular system on the silver data: a T5-base claim extractor, an E5-base dense retriever trained on about 3.2M ⟨claim, evidence⟩ pairs, and a DeBERTa-v3-large NLI claim verifier trained on 3.2M triplets. A text counts as factual only if every extracted claim is verified against the retrieved passages. Fine-tuning lifts the retriever’s Recall@30 from 0.52 to 0.95.

Results

The metric is balanced accuracy (%) on the gold benchmark, so a random guess scores 50. Open-weight models are averaged over five seeds. “E2E” is Task 1, “claim verif.” is Task 2.

Sources: Tables 5, 6 and 7. The fine-tuned row is the full 1B pipeline on Task 1 and the 0.4B claim verifier alone on Task 2. “Best prompt” is the highest of zero-shot, few-shot and their explain-then-answer variants.

What the numbers say:

• End-to-end factuality evaluation is hard. Open-weight LLMs score between 51.2 and 59.2 zero-shot. The best open-weight score in any prompt setting is 61.7 (Llama-3.3-70B-Instruct, few-shot, Table 4).
• Retrieval mostly does not help LLMs. With the top 30 retrieved passages, most models stay flat or drop. GPT-4o is the exception, going from 60.8 to 68.0. Truncating the RAG prompt from 18k down to 2k tokens leaves performance roughly stable (Figure 3), so long context alone is not the cause.
• A small specialised pipeline wins. The 1B pipeline trained on LLM-Oasis reaches 69.24, above every LLM tested, including GPT-4o with retrieval.
• Claim verification is much easier than end-to-end evaluation. With the right evidence given and a single claim of about 100 tokens of context (against about 4k in the end-to-end task), best-prompt scores rise to between 74.98 and 94.08. The 0.4B verifier reaches 93.30, close to Llama-3.3-70B (94.08) and GPT-4o (93.93) in their best settings.

Limitations

• The training data is LLM-generated. Human checks show 89 to 99% accuracy per step, but errors remain. The authors recommend the silver data for building systems and the gold benchmark for evaluation.
• One prompt for all four steps. Generation used a single GPT-4 call per passage instead of multi-step prompting, mainly to save cost (multi-step would have cost about four times as much). A qualitative comparison showed no clear quality gain from multi-step.
• Wikipedia changes. Unfactual texts stay unfactual, but factual texts can become outdated as pages are revised.
• Popular pages only. Long-tail facts are not targeted, though popular pages still contain lesser-known details.
• English only. Other languages and domains are left to future work.
• Non-commercial license. CC BY-NC-SA 4.0.

Questions people ask

How is LLM-Oasis different from FEVER? FEVER has 185,445 human-written claims built from about 10k Wikipedia pages and asks you to verify one claim at a time. LLM-Oasis asks whether a multi-sentence text is factual, with the error hidden in one of several claims, and covers 81,275 pages. The falsification prompt rules out simple negations and changes to dates, numbers or names, aiming for plausible substitutions rather than flipped verbs.

How is it different from FACTOR or FELM? Both are benchmarks: FACTOR has 4,266 instances and FELM 847 LLM outputs. LLM-Oasis is large enough to train factuality evaluators, with 81k text pairs and 681,201 claims, and it is not tied to one task such as summarisation.

What tasks does it support? Two benchmark tasks: end-to-end factuality evaluation of raw text (Task 1) and evidence-based claim verification (Task 2). The four training datasets also cover claim extraction, claim falsification and factual or unfactual text generation as separate supervised tasks.

How do I evaluate my LLM on it? For Task 1, predict True or False for each text in Babelscape/LLM-Oasis_e2e_factuality_evaluation (not publicly accessible on the Hub at the time of writing) and submit the .jsonl through the form in the README; results are reported as balanced accuracy. For Task 2, the gold claim-evidence pairs are public in Babelscape/LLM-Oasis_claim_verification, also without labels; the README does not describe a separate submission route.

Can I train on it? Yes. The four silver datasets have train and validation splits, and the paper’s baseline fine-tunes T5, E5 and DeBERTa models on them. Use is limited to non-commercial purposes under CC BY-NC-SA 4.0.

Does giving the model Wikipedia evidence solve the task? Not for most LLMs. At 18k tokens the passage needed to verify a text is in the retrieved evidence 95% of the time, yet most open-weight models score about the same as without it. The difficulty is using the evidence, not finding it.

Cite

@article{10.1162/COLI.a.575,
    author = {Scirè, Alessandro and Bejgu, Andrei Stefan and Tedeschi, Simone and Ghonim, Karim and Martelli, Federico and Navigli, Roberto},
    title = {Truth or Mirage? Towards End-To-End Factuality Evaluation with LLM-Oasis},
    journal = {Computational Linguistics},
    volume = {52},
    number = {1},
    pages = {1-41},
    year = {2026},
    issn = {0891-2017},
    doi = {10.1162/COLI.a.575},
    url = {https://doi.org/10.1162/COLI.a.575}
}

Paper: MIT Press · arXiv · Code: GitHub · Data: Hugging Face collection