107 lines
3.5 KiB
Groff
107 lines
3.5 KiB
Groff
.TH CONVERT_HF_TO_GGUF_UPDATE 1 "November 2026" "nvfp4_gguf_quant" "User Commands"
|
|
.SH NAME
|
|
convert_hf_to_gguf_update \- update pre-tokenizer hashes and vocab tests for convert_hf_to_gguf
|
|
.SH SYNOPSIS
|
|
.B convert_hf_to_gguf_update
|
|
[\fB\-\-full\fR]
|
|
[\fB\-\-check\-missing\fR]
|
|
[\fB\-h\fR|\fB\-\-help\fR]
|
|
[\fIhf_token\fR]
|
|
.SH DESCRIPTION
|
|
.B convert_hf_to_gguf_update
|
|
is a developer and contributor utility for \fBllama.cpp\fR that analyzes Hugging Face
|
|
model tokenizers and updates the pre-tokenizer recognition logic in
|
|
\fIconversion/base.py\fR (specifically the \fBget_vocab_base_pre()\fR function).
|
|
.PP
|
|
Proper pre-tokenization is critical for inference correctness in \fBllama.cpp\fR. Different
|
|
model families preprocess whitespace, digits, punctuation, and Unicode characters
|
|
differently before applying byte-pair or subword merges.
|
|
.B convert_hf_to_gguf_update
|
|
tokenizes a standardized, multilingual benchmark string (\fBCHK_TXT\fR) across a curated
|
|
registry of models, computes SHA-256 hashes of the resulting token sequences, and maps
|
|
those hashes to recognized pre-tokenizer types:
|
|
.IP \(bu 2
|
|
\fBSPM\fR: SentencePiece
|
|
.IP \(bu 2
|
|
\fBBPE\fR: Byte-Pair Encoding (e.g. GPT-2, LLaMA 3, Qwen2, DeepSeek, Falcon)
|
|
.IP \(bu 2
|
|
\fBWPM\fR: WordPiece (e.g. BERT, BGE)
|
|
.IP \(bu 2
|
|
\fBUGM\fR: Unigram (e.g. T5)
|
|
.PP
|
|
In addition to updating source definitions, the script generates tokenizer test vectors
|
|
(\fI.inp\fR input text and \fI.out\fR expected token ID files) under \fImodels/\fR for validation
|
|
against \fBllama.cpp\fR's native test harness (\fItest-tokenizer-0\fR).
|
|
.SH POSITIONAL ARGUMENTS
|
|
.TP
|
|
.I hf_token
|
|
Optional Hugging Face user access token. If omitted, the script attempts to read the
|
|
cached token from \fI~/.cache/huggingface/token\fR.
|
|
.SH OPTIONS
|
|
.TP
|
|
.B \-\-full
|
|
Download the complete list of registered models in the script's registry.
|
|
Ensure that the provided Hugging Face token has access permissions to all gated models
|
|
(e.g. LLaMA, Gemma).
|
|
.TP
|
|
.B \-\-check\-missing
|
|
Only check for models in the registry that currently lack pre-tokenizer hash entries in
|
|
\fIconversion/base.py\fR without redownloading the full dataset. Ignored if \fB\-\-full\fR
|
|
is also passed.
|
|
.TP
|
|
.BR \-h ", " \-\-help
|
|
Display command-line help summary and exit.
|
|
.SH FILES
|
|
.TP
|
|
.I conversion/base.py
|
|
The target source file updated with new tokenizer hash mappings in
|
|
\fBget_vocab_base_pre()\fR.
|
|
.TP
|
|
.I ~/.cache/huggingface/token
|
|
Default fallback path used to authenticate against Hugging Face Hub.
|
|
.TP
|
|
.I models/tokenizers/<name>/
|
|
Local cache directory where downloaded tokenizer files are stored.
|
|
.TP
|
|
.I models/ggml-vocab-<name>.gguf.inp
|
|
Input test strings written for llama.cpp tokenization testing.
|
|
.TP
|
|
.I models/ggml-vocab-<name>.gguf.out
|
|
Reference token IDs produced by Hugging Face \fBAutoTokenizer\fR for regression tests.
|
|
.SH EXAMPLES
|
|
.TP
|
|
.B Check for newly added models missing pre-tokenizer signatures:
|
|
.RS
|
|
.nf
|
|
convert_hf_to_gguf_update --check-missing
|
|
.fi
|
|
.RE
|
|
.TP
|
|
.B Run full update with explicit Hugging Face access token:
|
|
.RS
|
|
.nf
|
|
convert_hf_to_gguf_update --full hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
|
.fi
|
|
.RE
|
|
.TP
|
|
.B Generate vocabulary testing files for a newly registered tokenizer:
|
|
.RS
|
|
.nf
|
|
python3 convert_hf_to_gguf.py models/tokenizers/llama-bpe/ \\
|
|
--outfile models/ggml-vocab-llama-bpe.gguf --vocab-only
|
|
.fi
|
|
.RE
|
|
.SH EXIT STATUS
|
|
.TP
|
|
.B 0
|
|
Success.
|
|
.TP
|
|
.B 1
|
|
An error occurred (e.g. network failure, invalid token, or missing target files).
|
|
.SH SEE ALSO
|
|
.BR convert_hf_to_gguf (1),
|
|
.BR convert_lora_to_gguf (1),
|
|
.BR convert_llama_ggml_to_gguf (1),
|
|
.BR gguf_info (1)
|
|
|