.TH CONVERT_HF_TO_GGUF_UPDATE 1 "November 2026" "nvfp4_gguf_quant" "User Commands"
.SH NAME
convert_hf_to_gguf_update \- update pre-tokenizer hashes and vocab tests for convert_hf_to_gguf
.SH SYNOPSIS
.B convert_hf_to_gguf_update
[\fB\-\-full\fR]
[\fB\-\-check\-missing\fR]
[\fB\-h\fR|\fB\-\-help\fR]
[\fIhf_token\fR]
.SH DESCRIPTION
.B convert_hf_to_gguf_update
is a developer and contributor utility for \fBllama.cpp\fR that analyzes Hugging Face
model tokenizers and updates the pre-tokenizer recognition logic in
\fIconversion/base.py\fR (specifically the \fBget_vocab_base_pre()\fR function).
.PP
Proper pre-tokenization is critical for inference correctness in \fBllama.cpp\fR. Different
model families preprocess whitespace, digits, punctuation, and Unicode characters
differently before applying byte-pair or subword merges.
.B convert_hf_to_gguf_update
tokenizes a standardized, multilingual benchmark string (\fBCHK_TXT\fR) across a curated
registry of models, computes SHA-256 hashes of the resulting token sequences, and maps
those hashes to recognized pre-tokenizer types:
.IP \(bu 2
\fBSPM\fR: SentencePiece
.IP \(bu 2
\fBBPE\fR: Byte-Pair Encoding (e.g. GPT-2, LLaMA 3, Qwen2, DeepSeek, Falcon)
.IP \(bu 2
\fBWPM\fR: WordPiece (e.g. BERT, BGE)
.IP \(bu 2
\fBUGM\fR: Unigram (e.g. T5)
.PP
In addition to updating source definitions, the script generates tokenizer test vectors
(\fI.inp\fR input text and \fI.out\fR expected token ID files) under \fImodels/\fR for validation
against \fBllama.cpp\fR's native test harness (\fItest-tokenizer-0\fR).
.SH POSITIONAL ARGUMENTS
.TP
.I hf_token
Optional Hugging Face user access token. If omitted, the script attempts to read the
cached token from \fI~/.cache/huggingface/token\fR.
.SH OPTIONS
.TP
.B \-\-full
Download the complete list of registered models in the script's registry.
Ensure that the provided Hugging Face token has access permissions to all gated models
(e.g. LLaMA, Gemma).
.TP
.B \-\-check\-missing
Only check for models in the registry that currently lack pre-tokenizer hash entries in
\fIconversion/base.py\fR without redownloading the full dataset. Ignored if \fB\-\-full\fR
is also passed.
.TP
.BR \-h ", " \-\-help
Display command-line help summary and exit.
.SH FILES
.TP
.I conversion/base.py
The target source file updated with new tokenizer hash mappings in
\fBget_vocab_base_pre()\fR.
.TP
.I ~/.cache/huggingface/token
Default fallback path used to authenticate against Hugging Face Hub.
.TP
.I models/tokenizers/<name>/
Local cache directory where downloaded tokenizer files are stored.
.TP
.I models/ggml-vocab-<name>.gguf.inp
Input test strings written for llama.cpp tokenization testing.
.TP
.I models/ggml-vocab-<name>.gguf.out
Reference token IDs produced by Hugging Face \fBAutoTokenizer\fR for regression tests.
.SH EXAMPLES
.TP
.B Check for newly added models missing pre-tokenizer signatures:
.RS
.nf
convert_hf_to_gguf_update --check-missing
.fi
.RE
.TP
.B Run full update with explicit Hugging Face access token:
.RS
.nf
convert_hf_to_gguf_update --full hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
.fi
.RE
.TP
.B Generate vocabulary testing files for a newly registered tokenizer:
.RS
.nf
python3 convert_hf_to_gguf.py models/tokenizers/llama-bpe/ \\
    --outfile models/ggml-vocab-llama-bpe.gguf --vocab-only
.fi
.RE
.SH EXIT STATUS
.TP
.B 0
Success.
.TP
.B 1
An error occurred (e.g. network failure, invalid token, or missing target files).
.SH SEE ALSO
.BR convert_hf_to_gguf (1),
.BR convert_lora_to_gguf (1),
.BR convert_llama_ggml_to_gguf (1),
.BR gguf_info (1)