add scripts/manpages for llama.cpp converters
This commit is contained in:
zefie committed
1 parent
66d47eea6a
commit
0ef586d131
12 files changed
+667
No files matched your search
@@ -0,0 +1,266 @@
|
||||
.TH CONVERT_HF_TO_GGUF 1 "November 2026" "nvfp4_gguf_quant" "User Commands"
|
||||
.SH NAME
|
||||
convert_hf_to_gguf \- convert Hugging Face transformer models to GGUF format
|
||||
.SH SYNOPSIS
|
||||
.B convert_hf_to_gguf
|
||||
[\fB\-\-vocab\-only\fR]
|
||||
[\fB\-\-outfile\fR \fIOUTFILE\fR]
|
||||
[\fB\-\-outtype\fR {\fBf32\fR|\fBf16\fR|\fBbf16\fR|\fBq8_0\fR|\fBtq1_0\fR|\fBtq2_0\fR|\fBauto\fR}]
|
||||
[\fB\-\-bigendian\fR]
|
||||
[\fB\-\-use\-temp\-file\fR]
|
||||
[\fB\-\-no\-lazy\fR]
|
||||
[\fB\-\-model\-name\fR \fINAME\fR]
|
||||
[\fB\-\-split\-max\-tensors\fR \fIN\fR]
|
||||
[\fB\-\-split\-max\-size\fR \fISIZE\fR]
|
||||
[\fB\-\-dry\-run\fR]
|
||||
[\fB\-\-no\-tensor\-first\-split\fR]
|
||||
[\fB\-\-metadata\fR \fIPATH\fR]
|
||||
[\fB\-\-remote\fR]
|
||||
[\fB\-\-mmproj\fR]
|
||||
[\fB\-\-mtp\fR|\fB\-\-no\-nextn\fR|\fB\-\-dspark\fR]
|
||||
[\fB\-\-mistral\-format\fR]
|
||||
[\fB\-\-disable\-mistral\-community\-chat\-template\fR]
|
||||
[\fB\-\-sentence\-transformers\-dense\-modules\fR]
|
||||
[\fB\-\-fuse\-gate\-up\-exps\fR]
|
||||
[\fB\-\-fp8\-as\-q8\fR]
|
||||
[\fB\-\-fuse\-qkv\fR]
|
||||
[\fB\-\-target\-model\-dir\fR \fIDIR\fR]
|
||||
[\fB\-\-verbose\fR]
|
||||
[\fB\-h\fR|\fB\-\-help\fR]
|
||||
\fImodel\fR
|
||||
.PP
|
||||
.B convert_hf_to_gguf
|
||||
\fB\-\-print\-supported\-models\fR
|
||||
.SH DESCRIPTION
|
||||
.B convert_hf_to_gguf
|
||||
converts Hugging Face format model checkpoints (safetensors, PyTorch bin, config,
|
||||
and tokenizer files) into standard GGUF files compatible with \fBllama.cpp\fR and
|
||||
downstream tools such as Ollama.
|
||||
.PP
|
||||
The script parses architecture hyperparameters, maps model-specific tensor names
|
||||
to canonical GGML tensor names, handles tokenizer extraction and conversion (SPM,
|
||||
BPE, WPM, and UGM), encodes chat templates, and quantizes or casts weights into
|
||||
the requested precision format.
|
||||
.PP
|
||||
It also supports advanced operations including sharded split outputs, vision
|
||||
projector (mmproj) extraction, speculative decoding draft head extraction (MTP and
|
||||
DSpark), remote streaming from Hugging Face Hub without full checkpoint downloads,
|
||||
and Mixture-of-Experts (MoE) tensor fusion.
|
||||
.SH POSITIONAL ARGUMENTS
|
||||
.TP
|
||||
.I model
|
||||
Path to a local directory containing model configuration and tensor files, or
|
||||
a Hugging Face repository ID (e.g. \fIHuggingFaceTB/SmolLM2-1.7B-Instruct\fR) when
|
||||
used in combination with \fB\-\-remote\fR. Optional only when
|
||||
\fB\-\-print\-supported\-models\fR is specified.
|
||||
.SH OPTIONS
|
||||
.SS Output & Quantization
|
||||
.TP
|
||||
.BI \-\-outfile " " OUTFILE
|
||||
Destination path for the converted GGUF file. Defaults to an automatically generated
|
||||
path based on the model directory name and output type. If the pattern \fI{ftype}\fR
|
||||
is present in the filename, it is dynamically replaced with the chosen output type string.
|
||||
.TP
|
||||
.BI \-\-outtype " " TYPE
|
||||
Output tensor precision and format. Choices are:
|
||||
.RS
|
||||
.TP
|
||||
.B auto
|
||||
(Default) Preserves natively supported FP8 weights when present, and otherwise
|
||||
selects the highest-fidelity 16-bit float representation (BF16 or F16) matching the
|
||||
source model.
|
||||
.TP
|
||||
.B f32
|
||||
Convert all weight tensors to full 32-bit floating point.
|
||||
.TP
|
||||
.B f16
|
||||
Convert floating-point tensors to 16-bit IEEE half precision.
|
||||
.TP
|
||||
.B bf16
|
||||
Convert floating-point tensors to Brain Floating Point 16-bit.
|
||||
.TP
|
||||
.B q8_0
|
||||
Quantize non-1D weight tensors to 8-bit integer quantization (Q8_0).
|
||||
.TP
|
||||
.BR tq1_0 ", " tq2_0
|
||||
Convert weights to ternary quantized format.
|
||||
.RE
|
||||
.TP
|
||||
.B \-\-fp8\-as\-q8
|
||||
Store tensors dequantized from FP8 input weights as \fBQ8_0\fR instead of
|
||||
dequantizing all the way to BF16 or F16, saving memory and disk space.
|
||||
.SS Model & Vocabulary Selection
|
||||
.TP
|
||||
.B \-\-vocab\-only
|
||||
Extract and serialize only the model vocabulary, tokenizer merges, and metadata,
|
||||
omitting all neural network weight tensors.
|
||||
.TP
|
||||
.BI \-\-model\-name " " NAME
|
||||
Explicitly specify or override the model name stored in the GGUF general metadata.
|
||||
.TP
|
||||
.BI \-\-metadata " " PATH
|
||||
Path to a JSON file containing authorship and licensing metadata overrides to inject
|
||||
into the GGUF header.
|
||||
.TP
|
||||
.B \-\-print\-supported\-models
|
||||
Print all registered Hugging Face model architectures supported by the conversion
|
||||
framework and exit immediately.
|
||||
.SS Remote & Vision Modes
|
||||
.TP
|
||||
.B \-\-remote
|
||||
(Experimental) Stream safetensors index and tensor byte ranges directly from Hugging
|
||||
Face Hub over HTTP using partial range requests without downloading entire multi-gigabyte
|
||||
checkpoints to local disk. Only configuration and tokenizer files are downloaded. The
|
||||
\fImodel\fR argument must be a valid Hugging Face repository ID. For private or gated
|
||||
repositories, the \fBHF_TOKEN\fR environment variable must be set.
|
||||
.TP
|
||||
.B \-\-mmproj
|
||||
Extract and export only the multimodal projector (\fImmproj\fR) weights for vision-language
|
||||
models (e.g. LLaVA, Pixtral). The output filename is automatically prefixed with
|
||||
\fImmproj-\fR.
|
||||
.SS Speculative Decoding Draft Heads
|
||||
.TP
|
||||
.B \-\-mtp
|
||||
Export only the multi-token prediction (MTP) head as an independent GGUF model,
|
||||
suitable for use as a speculative decoding draft model. The output filename is
|
||||
prefixed with \fImtp-\fR. Mutually exclusive with \fB\-\-no\-nextn\fR and \fB\-\-dspark\fR.
|
||||
.TP
|
||||
.BR \-\-no\-nextn ", " \-\-no\-mtp
|
||||
Exclude NextN speculative draft tensors from the converted GGUF model. Typically paired
|
||||
with a secondary run using \fB\-\-mtp\fR or \fB\-\-dspark\fR to distribute target and
|
||||
draft heads in separate files.
|
||||
.TP
|
||||
.B \-\-dspark
|
||||
Export only DeepSeek-V4 or Mistral Large 4 DSpark speculative draft tensors as a
|
||||
separate GGUF file. Mutually exclusive with \fB\-\-mtp\fR and \fB\-\-no\-nextn\fR.
|
||||
.TP
|
||||
.BI \-\-target\-model\-dir " " DIR
|
||||
Path to the base target model directory. Required when converting standalone draft
|
||||
models (such as EAGLE3 or DFlash) that require tokenizer, hidden dimension, and layer
|
||||
count metadata from the primary model to construct a complete GGUF container.
|
||||
.SS Architecture & Tensor Adjustments
|
||||
.TP
|
||||
.B \-\-mistral\-format
|
||||
Indicate that the source model directory follows native Mistral format rather than
|
||||
standard Hugging Face Transformers conventions. Requires the \fBmistral\-common\fR
|
||||
Python package.
|
||||
.TP
|
||||
.B \-\-disable\-mistral\-community\-chat\-template
|
||||
Disable the fallback Hugging Face community chat template for Mistral models and use
|
||||
the official \fBmistral\-common\fR tokenizer and template engine.
|
||||
.TP
|
||||
.B \-\-sentence\-transformers\-dense\-modules
|
||||
Include dense projection modules from sentence-transformers embeddings models
|
||||
(e.g. \fIgoogle/embeddinggemma-300m\fR). By default, these modules are omitted.
|
||||
.TP
|
||||
.B \-\-fuse\-gate\-up\-exps
|
||||
Fuse separate MoE expert tensors \fIgate_exps\fR and \fIup_exps\fR into a unified
|
||||
\fIgate_up_exps\fR tensor for improved memory access and inference performance.
|
||||
.TP
|
||||
.B \-\-fuse\-qkv
|
||||
Fuse separate query, key, and value projection weight matrices into a single contiguous
|
||||
QKV tensor where supported by the architecture.
|
||||
.SS Sharding & Splitting
|
||||
.TP
|
||||
.BI \-\-split\-max\-tensors " " N
|
||||
Maximum number of tensors per output GGUF shard. Defaults to 0 (splitting disabled).
|
||||
.TP
|
||||
.BI \-\-split\-max\-size " " SIZE
|
||||
Maximum byte size per output GGUF shard. Can be specified with unit suffixes \fBK\fR,
|
||||
\fBM\fR, or \fBG\fR (e.g. \fB4G\fR or \fB2000M\fR). Defaults to 0 (no size-based splitting).
|
||||
.TP
|
||||
.B \-\-no\-tensor\-first\-split
|
||||
Do not store any model weight tensors in the first shard (shard 00001). Shard 1 will
|
||||
contain only metadata, key-values, and vocabulary.
|
||||
.TP
|
||||
.B \-\-dry\-run
|
||||
Simulate the conversion and splitting plan, logging projected shard sizes and tensor
|
||||
distributions without writing any output files to disk.
|
||||
.SS System & Execution
|
||||
.TP
|
||||
.B \-\-use\-temp\-file
|
||||
Write intermediate tensors to temporary scratch files on disk rather than holding them
|
||||
in RAM. Recommended on systems with constrained system memory to prevent out-of-memory
|
||||
process termination. Cannot be used together with sharding.
|
||||
.TP
|
||||
.B \-\-no\-lazy
|
||||
Disable lazy tensor evaluation, forcing all tensors to be computed and staged in RAM
|
||||
prior to writing. Useful if lazy evaluation encounters issues.
|
||||
.TP
|
||||
.B \-\-bigendian
|
||||
Target big-endian architectures when serializing tensor headers and raw data.
|
||||
.TP
|
||||
.B \-\-verbose
|
||||
Enable verbose debug logging output.
|
||||
.TP
|
||||
.BR \-h ", " \-\-help
|
||||
Display command-line syntax and option descriptions, then exit.
|
||||
.SH ENVIRONMENT
|
||||
.TP
|
||||
.B HF_TOKEN
|
||||
Hugging Face user authentication token. Required when accessing gated or private
|
||||
models in conjunction with \fB\-\-remote\fR.
|
||||
.TP
|
||||
.B NO_LOCAL_GGUF
|
||||
When set in the environment, prevents the script from automatically prepending the
|
||||
local repository's \fIgguf\-py\fR subdirectory to Python's module search path.
|
||||
.SH EXAMPLES
|
||||
.TP
|
||||
.B Convert a local Hugging Face model to default 16-bit GGUF:
|
||||
.RS
|
||||
.nf
|
||||
convert_hf_to_gguf ./Meta-Llama-3-8B-Instruct
|
||||
.fi
|
||||
.RE
|
||||
.TP
|
||||
.B Convert directly to 8-bit integer quantization with explicit output name:
|
||||
.RS
|
||||
.nf
|
||||
convert_hf_to_gguf --outtype q8_0 --outfile ./llama3-8b-q8.gguf ./Meta-Llama-3-8B
|
||||
.fi
|
||||
.RE
|
||||
.TP
|
||||
.B Convert a remote Hugging Face repository without downloading weights to disk:
|
||||
.RS
|
||||
.nf
|
||||
export HF_TOKEN="hf_..."
|
||||
convert_hf_to_gguf --remote HuggingFaceTB/SmolLM2-1.7B-Instruct
|
||||
.fi
|
||||
.RE
|
||||
.TP
|
||||
.B Split a large 70B model into 4-gigabyte GGUF shards:
|
||||
.RS
|
||||
.nf
|
||||
convert_hf_to_gguf --split-max-size 4G --outtype f16 ./Meta-Llama-3-70B
|
||||
.fi
|
||||
.RE
|
||||
.TP
|
||||
.B Extract only the multimodal vision projector for a vision-language model:
|
||||
.RS
|
||||
.nf
|
||||
convert_hf_to_gguf --mmproj ./Qwen2-VL-7B-Instruct
|
||||
.fi
|
||||
.RE
|
||||
.TP
|
||||
.B Extract only the vocabulary and tokenizer metadata:
|
||||
.RS
|
||||
.nf
|
||||
convert_hf_to_gguf --vocab-only --outfile vocab.gguf ./Mistral-7B-v0.3
|
||||
.fi
|
||||
.RE
|
||||
.SH EXIT STATUS
|
||||
.TP
|
||||
.B 0
|
||||
Successful completion.
|
||||
.TP
|
||||
.B 1
|
||||
Conversion failed due to invalid arguments, missing dependencies, unsupported architecture,
|
||||
or I/O error.
|
||||
.SH SEE ALSO
|
||||
.BR convert_hf_to_gguf_update (1),
|
||||
.BR convert_lora_to_gguf (1),
|
||||
.BR convert_llama_ggml_to_gguf (1),
|
||||
.BR hf_to_nvfp4_gguf (1),
|
||||
.BR gguf_to_nvfp4_gguf (1),
|
||||
.BR gguf_info (1)
|
||||
|
||||
Reference in new issue
Block a user