nvfp4_gguf_quant/man/man1/convert_hf_to_gguf.1

.TH CONVERT_HF_TO_GGUF 1 "November 2026" "nvfp4_gguf_quant" "User Commands"
.SH NAME
convert_hf_to_gguf \- convert Hugging Face transformer models to GGUF format
.SH SYNOPSIS
.B convert_hf_to_gguf
[\fB\-\-vocab\-only\fR]
[\fB\-\-outfile\fR \fIOUTFILE\fR]
[\fB\-\-outtype\fR {\fBf32\fR|\fBf16\fR|\fBbf16\fR|\fBq8_0\fR|\fBtq1_0\fR|\fBtq2_0\fR|\fBauto\fR}]
[\fB\-\-bigendian\fR]
[\fB\-\-use\-temp\-file\fR]
[\fB\-\-no\-lazy\fR]
[\fB\-\-model\-name\fR \fINAME\fR]
[\fB\-\-split\-max\-tensors\fR \fIN\fR]
[\fB\-\-split\-max\-size\fR \fISIZE\fR]
[\fB\-\-dry\-run\fR]
[\fB\-\-no\-tensor\-first\-split\fR]
[\fB\-\-metadata\fR \fIPATH\fR]
[\fB\-\-remote\fR]
[\fB\-\-mmproj\fR]
[\fB\-\-mtp\fR|\fB\-\-no\-nextn\fR|\fB\-\-dspark\fR]
[\fB\-\-mistral\-format\fR]
[\fB\-\-disable\-mistral\-community\-chat\-template\fR]
[\fB\-\-sentence\-transformers\-dense\-modules\fR]
[\fB\-\-fuse\-gate\-up\-exps\fR]
[\fB\-\-fp8\-as\-q8\fR]
[\fB\-\-fuse\-qkv\fR]
[\fB\-\-target\-model\-dir\fR \fIDIR\fR]
[\fB\-\-verbose\fR]
[\fB\-h\fR|\fB\-\-help\fR]
\fImodel\fR
.PP
.B convert_hf_to_gguf
\fB\-\-print\-supported\-models\fR
.SH DESCRIPTION
.B convert_hf_to_gguf
converts Hugging Face format model checkpoints (safetensors, PyTorch bin, config,
and tokenizer files) into standard GGUF files compatible with \fBllama.cpp\fR and
downstream tools such as Ollama.
.PP
The script parses architecture hyperparameters, maps model-specific tensor names
to canonical GGML tensor names, handles tokenizer extraction and conversion (SPM,
BPE, WPM, and UGM), encodes chat templates, and quantizes or casts weights into
the requested precision format.
.PP
It also supports advanced operations including sharded split outputs, vision
projector (mmproj) extraction, speculative decoding draft head extraction (MTP and
DSpark), remote streaming from Hugging Face Hub without full checkpoint downloads,
and Mixture-of-Experts (MoE) tensor fusion.
.SH POSITIONAL ARGUMENTS
.TP
.I model
Path to a local directory containing model configuration and tensor files, or
a Hugging Face repository ID (e.g. \fIHuggingFaceTB/SmolLM2-1.7B-Instruct\fR) when
used in combination with \fB\-\-remote\fR. Optional only when
\fB\-\-print\-supported\-models\fR is specified.
.SH OPTIONS
.SS Output & Quantization
.TP
.BI \-\-outfile " " OUTFILE
Destination path for the converted GGUF file. Defaults to an automatically generated
path based on the model directory name and output type. If the pattern \fI{ftype}\fR
is present in the filename, it is dynamically replaced with the chosen output type string.
.TP
.BI \-\-outtype " " TYPE
Output tensor precision and format. Choices are:
.RS
.TP
.B auto
(Default) Preserves natively supported FP8 weights when present, and otherwise
selects the highest-fidelity 16-bit float representation (BF16 or F16) matching the
source model.
.TP
.B f32
Convert all weight tensors to full 32-bit floating point.
.TP
.B f16
Convert floating-point tensors to 16-bit IEEE half precision.
.TP
.B bf16
Convert floating-point tensors to Brain Floating Point 16-bit.
.TP
.B q8_0
Quantize non-1D weight tensors to 8-bit integer quantization (Q8_0).
.TP
.BR tq1_0 ", " tq2_0
Convert weights to ternary quantized format.
.RE
.TP
.B \-\-fp8\-as\-q8
Store tensors dequantized from FP8 input weights as \fBQ8_0\fR instead of
dequantizing all the way to BF16 or F16, saving memory and disk space.
.SS Model & Vocabulary Selection
.TP
.B \-\-vocab\-only
Extract and serialize only the model vocabulary, tokenizer merges, and metadata,
omitting all neural network weight tensors.
.TP
.BI \-\-model\-name " " NAME
Explicitly specify or override the model name stored in the GGUF general metadata.
.TP
.BI \-\-metadata " " PATH
Path to a JSON file containing authorship and licensing metadata overrides to inject
into the GGUF header.
.TP
.B \-\-print\-supported\-models
Print all registered Hugging Face model architectures supported by the conversion
framework and exit immediately.
.SS Remote & Vision Modes
.TP
.B \-\-remote
(Experimental) Stream safetensors index and tensor byte ranges directly from Hugging
Face Hub over HTTP using partial range requests without downloading entire multi-gigabyte
checkpoints to local disk. Only configuration and tokenizer files are downloaded. The
\fImodel\fR argument must be a valid Hugging Face repository ID. For private or gated
repositories, the \fBHF_TOKEN\fR environment variable must be set.
.TP
.B \-\-mmproj
Extract and export only the multimodal projector (\fImmproj\fR) weights for vision-language
models (e.g. LLaVA, Pixtral). The output filename is automatically prefixed with
\fImmproj-\fR.
.SS Speculative Decoding Draft Heads
.TP
.B \-\-mtp
Export only the multi-token prediction (MTP) head as an independent GGUF model,
suitable for use as a speculative decoding draft model. The output filename is
prefixed with \fImtp-\fR. Mutually exclusive with \fB\-\-no\-nextn\fR and \fB\-\-dspark\fR.
.TP
.BR \-\-no\-nextn ", " \-\-no\-mtp
Exclude NextN speculative draft tensors from the converted GGUF model. Typically paired
with a secondary run using \fB\-\-mtp\fR or \fB\-\-dspark\fR to distribute target and
draft heads in separate files.
.TP
.B \-\-dspark
Export only DeepSeek-V4 or Mistral Large 4 DSpark speculative draft tensors as a
separate GGUF file. Mutually exclusive with \fB\-\-mtp\fR and \fB\-\-no\-nextn\fR.
.TP
.BI \-\-target\-model\-dir " " DIR
Path to the base target model directory. Required when converting standalone draft
models (such as EAGLE3 or DFlash) that require tokenizer, hidden dimension, and layer
count metadata from the primary model to construct a complete GGUF container.
.SS Architecture & Tensor Adjustments
.TP
.B \-\-mistral\-format
Indicate that the source model directory follows native Mistral format rather than
standard Hugging Face Transformers conventions. Requires the \fBmistral\-common\fR
Python package.
.TP
.B \-\-disable\-mistral\-community\-chat\-template
Disable the fallback Hugging Face community chat template for Mistral models and use
the official \fBmistral\-common\fR tokenizer and template engine.
.TP
.B \-\-sentence\-transformers\-dense\-modules
Include dense projection modules from sentence-transformers embeddings models
(e.g. \fIgoogle/embeddinggemma-300m\fR). By default, these modules are omitted.
.TP
.B \-\-fuse\-gate\-up\-exps
Fuse separate MoE expert tensors \fIgate_exps\fR and \fIup_exps\fR into a unified
\fIgate_up_exps\fR tensor for improved memory access and inference performance.
.TP
.B \-\-fuse\-qkv
Fuse separate query, key, and value projection weight matrices into a single contiguous
QKV tensor where supported by the architecture.
.SS Sharding & Splitting
.TP
.BI \-\-split\-max\-tensors " " N
Maximum number of tensors per output GGUF shard. Defaults to 0 (splitting disabled).
.TP
.BI \-\-split\-max\-size " " SIZE
Maximum byte size per output GGUF shard. Can be specified with unit suffixes \fBK\fR,
\fBM\fR, or \fBG\fR (e.g. \fB4G\fR or \fB2000M\fR). Defaults to 0 (no size-based splitting).
.TP
.B \-\-no\-tensor\-first\-split
Do not store any model weight tensors in the first shard (shard 00001). Shard 1 will
contain only metadata, key-values, and vocabulary.
.TP
.B \-\-dry\-run
Simulate the conversion and splitting plan, logging projected shard sizes and tensor
distributions without writing any output files to disk.
.SS System & Execution
.TP
.B \-\-use\-temp\-file
Write intermediate tensors to temporary scratch files on disk rather than holding them
in RAM. Recommended on systems with constrained system memory to prevent out-of-memory
process termination. Cannot be used together with sharding.
.TP
.B \-\-no\-lazy
Disable lazy tensor evaluation, forcing all tensors to be computed and staged in RAM
prior to writing. Useful if lazy evaluation encounters issues.
.TP
.B \-\-bigendian
Target big-endian architectures when serializing tensor headers and raw data.
.TP
.B \-\-verbose
Enable verbose debug logging output.
.TP
.BR \-h ", " \-\-help
Display command-line syntax and option descriptions, then exit.
.SH ENVIRONMENT
.TP
.B HF_TOKEN
Hugging Face user authentication token. Required when accessing gated or private
models in conjunction with \fB\-\-remote\fR.
.TP
.B NO_LOCAL_GGUF
When set in the environment, prevents the script from automatically prepending the
local repository's \fIgguf\-py\fR subdirectory to Python's module search path.
.SH EXAMPLES
.TP
.B Convert a local Hugging Face model to default 16-bit GGUF:
.RS
.nf
convert_hf_to_gguf ./Meta-Llama-3-8B-Instruct
.fi
.RE
.TP
.B Convert directly to 8-bit integer quantization with explicit output name:
.RS
.nf
convert_hf_to_gguf --outtype q8_0 --outfile ./llama3-8b-q8.gguf ./Meta-Llama-3-8B
.fi
.RE
.TP
.B Convert a remote Hugging Face repository without downloading weights to disk:
.RS
.nf
export HF_TOKEN="hf_..."
convert_hf_to_gguf --remote HuggingFaceTB/SmolLM2-1.7B-Instruct
.fi
.RE
.TP
.B Split a large 70B model into 4-gigabyte GGUF shards:
.RS
.nf
convert_hf_to_gguf --split-max-size 4G --outtype f16 ./Meta-Llama-3-70B
.fi
.RE
.TP
.B Extract only the multimodal vision projector for a vision-language model:
.RS
.nf
convert_hf_to_gguf --mmproj ./Qwen2-VL-7B-Instruct
.fi
.RE
.TP
.B Extract only the vocabulary and tokenizer metadata:
.RS
.nf
convert_hf_to_gguf --vocab-only --outfile vocab.gguf ./Mistral-7B-v0.3
.fi
.RE
.SH EXIT STATUS
.TP
.B 0
Successful completion.
.TP
.B 1
Conversion failed due to invalid arguments, missing dependencies, unsupported architecture,
or I/O error.
.SH SEE ALSO
.BR convert_hf_to_gguf_update (1),
.BR convert_lora_to_gguf (1),
.BR convert_llama_ggml_to_gguf (1),
.BR hf_to_nvfp4_gguf (1),
.BR gguf_to_nvfp4_gguf (1),
.BR gguf_info (1)