Vocabulary adaptation for an Italian LLM: cutting token fertility with SAVA (NAACL 2025 Findings)

English-first LLMs split Italian words into more pieces than English words, so every Italian prompt costs more tokens, more memory and more latency. In this paper we replace the tokenizer of Mistral-7B-v0.1 and Llama-3.1-8B with the Italian-English tokenizer of the Minerva models, initialise the new embeddings with a method we call SAVA (Semantic Alignment Vocabulary Adaptation), and continue training on 12B tokens. Italian token fertility drops by 25% for Mistral and 16% for Llama, Llama-3.1-8B shrinks from 8.03B to 7.25B parameters, and the adapted models recover base-model quality on Italian tasks.

What is token fertility and why it costs you

Token fertility is the average number of tokens a tokenizer produces per word. A tokenizer trained mostly on English learns long English merges and few Italian ones, so “aggiornamento” ends up in several pieces. Higher fertility means longer sequences for the same text: more compute per sentence, less text in the context window, slower generation.

Table 2 of the paper measures it on Italian and English text from CulturaX and Wikipedia:

The Minerva tokenizer also costs nothing in model size for Mistral (32,000 to 32,768 entries, 7.24B to 7.25B parameters) and saves a lot for Llama-3.1-8B (128,256 to 32,768 entries, a 75% smaller vocabulary, 8.03B to 7.25B parameters, Table 1). That is the efficiency gain. The open question is how to give the model useful embeddings for tokens it has never seen.

How to adapt an LLM’s vocabulary to a new language

1. Measure token fertility

The repo’s embedding_analysis/compute_fertility.py averages len(tokens) / len(words) per document over 50,000 documents, splitting words with NLTK word_tokenize. This shorter version uses whitespace words, so its numbers will not match Table 2 exactly. Both tokenizers below are public: the SAVA-adapted Mistral ships the Minerva tokenizer.

from itertools import islice
from datasets import load_dataset
from transformers import AutoTokenizer

docs = [d["text"] for d in islice(
    load_dataset("wikimedia/wikipedia", "20231101.it", split="train", streaming=True), 2000)]

def fertility(tok_id: str) -> float:
    tok = AutoTokenizer.from_pretrained(tok_id)
    per_doc = [len(tok(t, add_special_tokens=False).input_ids) / max(len(t.split()), 1) for t in docs]
    return sum(per_doc) / len(per_doc)

for tok_id in ["mistralai/Mistral-7B-v0.1", "SemanticAlignment/Mistral-v0.1-Italian-SAVA"]:
    print(f"{tok_id}: {fertility(tok_id):.2f} tokens per word")

2. Train a target-language tokenizer

The paper did not train a tokenizer: it reused Minerva’s, which was trained on Italian and English CulturaX. If no good tokenizer exists for your language, train_new_from_iterator keeps the algorithm and special tokens of an existing fast tokenizer and learns a new vocabulary. A mix of target and source language text keeps English usable.

from itertools import islice
from datasets import load_dataset
from transformers import AutoTokenizer

stream = load_dataset("uonlp/CulturaX", "it", split="train", streaming=True)  # gated: accept terms on the Hub

def batches(n_docs=1_000_000, size=1000):
    it = (d["text"] for d in islice(stream, n_docs))
    while batch := list(islice(it, size)):
        yield batch

base = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
it_tok = base.train_new_from_iterator(batches(), vocab_size=32_768)
it_tok.save_pretrained("tokenizer-it")

3. Initialise the new embeddings

Every method fills the new embedding matrix the same way for shared tokens (copy the source vector) and differs for the rest. FVT (Fast Vocabulary Transfer, Gee et al. 2022) re-tokenises each new token with the old tokenizer and averages the old embeddings of the pieces:

import torch

@torch.no_grad()
def fvt_init(src_emb: torch.Tensor, src_tok, tgt_tok) -> torch.Tensor:
    """src_emb: [V_src, d] (input embeddings or LM head). Returns [V_tgt, d]."""
    src_vocab = src_tok.get_vocab()
    out = torch.empty(len(tgt_tok), src_emb.shape[1], dtype=src_emb.dtype)
    for token, idx in tgt_tok.get_vocab().items():
        if token in src_vocab:                       # shared token: copy
            out[idx] = src_emb[src_vocab[token]]
            continue
        text = tgt_tok.convert_tokens_to_string([token])
        pieces = src_tok(text, add_special_tokens=False).input_ids
        out[idx] = src_emb[pieces].mean(0) if pieces else src_emb.mean(0)
    return out

Matching shared tokens needs care when the two tokenizers mark spaces differently. The repo maps Llama’s byte-level Ġ to the SentencePiece ▁ before comparing vocabularies (_map_llama_to_minerva in adaptation/core/average_naive.py and semantic_alignment.py).

SAVA uses a helper model that already speaks the target tokenizer: Minerva-3B, whose embeddings cover every token in the new vocabulary. It learns an affine map ϕ(x) = Wx + b from Minerva-3B’s embedding space to the source model’s, fitted to minimise the squared error on tokens the two vocabularies share, then maps each new token’s Minerva-3B embedding into the source space (Section 3.1, Equation 1). In the code (adaptation/core/semantic_alignment.py) the map is a latentis Translator: both spaces are standardised and L2-normalised, dimensions are matched by zero padding, and the sgd aligner is SGDAffineAligner(num_steps=1000, lr=1e-3). When the LM head is not tied, a second translator is fitted on the LM head weights.

The README’s documented entry point, for Llama-3.1-8B:

cd sava/adaptation
python main.py \
  --source_model_name_or_path meta-llama/Llama-3.1-8B \
  --source_tokenizer_name_or_path meta-llama/Llama-3.1-8B \
  --helper_model_name_or_path sapienzanlp/Minerva-3B-base-v1.0 \
  --target_tokenizer_name_or_path sapienzanlp/Minerva-3B-base-v1.0 \
  --seed 42 \
  --initialization_method semantic_alignment \
  --aligner sgd \
  --output_dir /path/to/output/model/dir/llama-3.1-italian-sava

Other values of --initialization_method are average_naive (FVT), clp and clp_random. Tip: also pass --anchor_selection full (use the whole shared vocabulary as anchors) or --num_anchors N, since the default anchor selection expects a count. If you would rather skip the CLI, you can call the initializer from Python:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from core import SemanticAlignmentEmbeddingInitializer  # run from sava/adaptation/

src = "mistralai/Mistral-7B-v0.1"
helper = "sapienzanlp/Minerva-3B-base-v1.0"  # gated: accept terms on the Hub
target_tok = AutoTokenizer.from_pretrained(helper)

model = SemanticAlignmentEmbeddingInitializer(
    source_model=AutoModelForCausalLM.from_pretrained(src, torch_dtype=torch.float16),
    source_tokenizer=AutoTokenizer.from_pretrained(src),
    helper_model=AutoModelForCausalLM.from_pretrained(helper, torch_dtype=torch.float16),
    target_tokenizer=target_tok,
    seed=42,
    tie_weights=False,        # Mistral and Llama-3.1 have an untied LM head
    aligner="sgd",            # also "matrix" (least squares) or "ortho" (Procrustes)
    anchor_selection="full",  # fit on every shared token; "random" samples num_anchors of them
)()
model.save_pretrained("mistral-it-sava")
target_tok.save_pretrained("mistral-it-sava")

4. Continual training after adaptation

The paper trained with llm-foundry v0.8.0 (configs in training/yamls/pretrain/). The snippet below is a minimal recipe, not the paper’s exact setup: same data mix and learning rate, Hugging Face Trainer instead of llm-foundry, and a batch size you would scale to your hardware.

import torch
from itertools import chain
from datasets import load_dataset, interleave_datasets
from transformers import (AutoModelForCausalLM, AutoTokenizer, DataCollatorForLanguageModeling,
                          Trainer, TrainingArguments)

path, seq_len = "mistral-it-sava", 2048
tok = AutoTokenizer.from_pretrained(path)
tok.pad_token = tok.pad_token or tok.eos_token
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16)

cx = lambda lang: load_dataset("uonlp/CulturaX", lang, split="train", streaming=True).select_columns(["text"])
mix = interleave_datasets([cx("it"), cx("en")], probabilities=[0.75, 0.25], seed=17)

def tokenize_and_pack(batch):
    ids = list(chain.from_iterable(tok(batch["text"]).input_ids))
    n = len(ids) // seq_len * seq_len
    return {"input_ids": [ids[i:i + seq_len] for i in range(0, n, seq_len)]}

train = mix.map(tokenize_and_pack, batched=True, remove_columns=["text"])

args = TrainingArguments(
    output_dir="ckpt-mistral-it-sava",
    max_steps=2000,                  # streaming datasets need max_steps
    per_device_train_batch_size=4,
    gradient_accumulation_steps=32,  # tokens per step = devices x 4 x 32 x 2048
    learning_rate=1e-5,
    lr_scheduler_type="cosine",
    warmup_steps=20,
    bf16=True,
    gradient_checkpointing=True,
    logging_steps=10,
    save_steps=200,
)
Trainer(model=model, args=args, train_dataset=train,
        data_collator=DataCollatorForLanguageModeling(tok, mlm=False)).train()

What the paper actually ran (Section 4.2, training/README.md, the two YAML files):

Neither the paper nor the repo covers instruction tuning of the adapted models; they are base models.

SAVA vs FVT and the other methods

• Random: new tokens drawn from a normal distribution with the mean and variance of the source embeddings. The floor.
• FVT (Gee et al., 2022): mean of the source embeddings of the old-tokenizer pieces. No helper model, no training.
• CLP (Ostendorff and Rehm, 2023): each new token is a weighted sum of shared-token source embeddings, weighted by cosine similarity measured in the helper model’s space.
• SAVA (this paper): an affine map from the helper’s embedding space to the source’s, learned on shared tokens and applied to new ones.
• LAPT: continual training of the base model with its original tokenizer. No fertility gain; the reference for how much quality vocabulary adaptation should recover.

FVT and SAVA both start much closer to useful than Random and CLP. SAVA’s advantage is that it uses a model that has actually learned the target-language tokens in context, instead of guessing from how the old tokenizer splits them.

Results

Italian multiple choice (0-shot, six benchmarks from the translated part of ITA-Bench, average accuracy):

Early is 200 steps for Mistral and 300 for Llama; end is 2000 and 3000 (Tables 3 and 4). Every adapted Mistral beats the original Mistral by the end. SAVA and FVT reach at 400 batches what Random reaches at the end, about 80% less training, and Mistral recovers base-model performance after about 2B tokens. For Llama, SAVA scores 74.15 on BoolQ against 69.35 for FVT and 71.77 for LAPT.

Generation (Tables 5 and 6: FLoRes 5-shot, COMET-22; SQuAD-it 2-shot, ROUGE-L):

SAVA has the best SQuAD-it score among the trained variants for both models and the best EN→IT translation, the direction that generates Italian.

Embedding structure. Using relative representations over 256 anchor tokens, SAVA’s embeddings are the closest to Minerva-3B’s (Table 7): 44.81 at step 0 and 45.33 after training for Mistral, against 41.10 and 42.84 for CLP, 33.65 and 35.30 for FVT, 29.68 and 31.67 for Random.

Limitations

• English drops. All adapted Mistral models score below the original on English benchmarks; for Llama, LAPT keeps English better, likely thanks to its larger vocabulary.
• Translated benchmarks. The multiple-choice sets were machine-translated with Tower-Instruct-v0.2, which can add noise.
• Web data only. Continual training used CulturaX alone.
• Two models, one language. Mistral-7B-v0.1 and Llama-3.1-8B, Italian only; Random and CLP were not run on Llama.
• Compute. About 50,000 A100 GPU hours in total, estimated at 8,640 kgCO2eq (Appendix C).

Questions people ask

What is SAVA? Semantic Alignment Vocabulary Adaptation: initialise the embeddings of new tokens by mapping a helper model’s embeddings (Minerva-3B) into the source model’s space with an affine map learned on the shared vocabulary.

How much does vocabulary adaptation reduce token fertility for Italian? With the Minerva tokenizer, by 25% for Mistral-7B-v0.1 and up to 16% for Llama-3.1-8B, on CulturaX and Wikipedia text.

Is SAVA better than FVT? They are close. SAVA has a lower training loss from the start, the best end-of-training Italian average for Mistral (66.99 vs 66.94), better BoolQ and SQuAD-it scores, and better English for both models. FVT needs no helper model.

Can I skip continual training? No. Initialisation makes the model usable at step 0, but every method needs continual training to recover; the paper used about 12B tokens and the gap to the base model closed after about 2B for Mistral.

Does it need a helper model? SAVA and CLP do: a model already trained with the target tokenizer. The paper also tested Minerva-350M and Minerva-1B as helpers and found helper size had little effect on the loss, while fitting the map on fewer shared tokens slowed convergence (Appendix B).

Where are the adapted models? On Hugging Face under SemanticAlignment, e.g. SemanticAlignment/Mistral-v0.1-Italian-SAVA and SemanticAlignment/Llama-3.1-8B-Italian-SAVA.

Cite

@inproceedings{moroni-etal-2025-optimizing,
    title = "Optimizing {LLM}s for {I}talian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation",
    author = "Moroni, Luca  and
      Puccetti, Giovanni  and
      Huguet Cabot, Pere-Llu{\'i}s  and
      Bejgu, Andrei Stefan  and
      Miaschi, Alessio  and
      Barba, Edoardo  and
      Dell{'}Orletta, Felice  and
      Esuli, Andrea  and
      Navigli, Roberto",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {NAACL} 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-naacl.371/",
    doi = "10.18653/v1/2025.findings-naacl.371",
    pages = "6661--6675",
    ISBN = "979-8-89176-195-7"
}

Paper: ACL Anthology · arXiv · Code: GitHub · Models: Mistral collection · Llama collection