.TH GGUF_TO_NVFP4_HF 1 "November 2026" "nvfp4_gguf_quant" "User Commands"
.SH NAME
gguf_to_nvfp4_hf \- convert and quantize GGUF models to NVFP4 Hugging Face format
.SH SYNOPSIS
.B gguf_to_nvfp4_hf
[\fB\-o\fR \fIOUTPUT\fR]
[\fB\-\-overwrite\fR]
[\fB\-\-format\fR \fIFORMAT\fR]
[\fB\-\-no\-quant\fR]
[\fB\-\-outtype\fR \fITYPE\fR]
[\fB\-\-output\-dtype\fR \fIDTYPE\fR]
[\fB\-\-hf\-template\fR \fIDIR\fR]
[\fB\-\-context\-length\fR \fILEN\fR]
[\fB\-\-model\-name\fR \fINAME\fR]
[\fB\-\-checkpoint\-dir\fR \fIDIR\fR]
[\fB\-\-fresh\fR]
[\fB\-\-keep\-checkpoints\fR]
[\fB\-\-eager\fR]
[\fB\-\-low\-memory\fR]
[\fB\-\-heur\fR]
[\fB\-\-verbose\fR]
[\fB\-\-self\-test\fR]
[\fB\-\-llama\-cpp\fR \fIPATH\fR]
[\fB\-\-convert\-to\-quant\fR \fIPATH\fR]
[\fB\-\-device\fR \fIDEVICE\fR]
[\fB\-\-simple\fR]
[\fB\-\-optimizer\fR \fIOPT\fR]
[\fB\-\-num\-iter\fR \fIN\fR]
[\fB\-\-lr\fR \fILR\fR]
[\fB\-\-lr\-patience\fR \fIN\fR]
[\fB\-\-lr\-min\fR \fIMIN\fR]
[\fB\-\-lr\-schedule\fR \fISCHED\fR]
[\fB\-\-scale\-optimization\fR \fIMODE\fR]
[\fB\-\-scale\-refinement\-rounds\fR \fIN\fR]
[\fB\-\-full\-matrix\fR]
[\fB\-\-top\-p\fR \fIP\fR]
[\fB\-\-min\-k\fR \fIK\fR]
[\fB\-\-max\-k\fR \fIK\fR]
[\fB\-\-early\-stop\-loss\fR \fILOSS\fR]
[\fB\-\-early\-stop\-stall\fR \fISTALL\fR]
[\fB\-\-target\-loss\fR \fITARGET\fR]
[\fB\-\-early\-stop\-floor\-stall\fR \fIFLOOR\fR]
[\fB\-\-calib\-samples\fR \fISAMPLES\fR]
[\fB\-\-seed\fR \fISEED\fR]
[\fB\-\-no\-bias\-correction\fR]
[\fB\-\-quantize\-output\fR]
[\fB\-\-exclude\fR \fIREGEX\fR]
[\fB\-h\fR|\fB\-\-help\fR]
\fIgguf\fR
.SH DESCRIPTION
.B gguf_to_nvfp4_hf
converts a GGUF model file into a standard Hugging Face model directory
(\fIconfig.json\fR + safetensors) with NVIDIA FP4 (NVFP4) quantized weights.
.PP
GGUF tensor names are reverse-mapped to canonical Hugging Face conventions
(e.g., \fIblk.0.attn_q.weight\fR maps to \fImodel.layers.0.self_attn.q_proj.weight\fR),
and RoPE permutations for Q and K are inverted.
.PP
If the source GGUF contains float or non-NVFP4 weights, 2D weight matrices are quantized
using convert_to_quant's learned rounding (or simple round-to-grid with \fB\-\-simple\fR).
If the source GGUF already contains NVFP4 super-blocks, they are unpacked and repacked
into ModelOpt-compatible safetensors without loss of precision.
.PP
If \fB\-\-no\-quant\fR is specified, the model is dequantized to standard float format
(\fI--outtype f32, f16, bf16, or auto\fR).
.PP
An optional reference Hugging Face model directory can be provided with \fB\-\-hf\-template\fR
to copy exact upstream tokenizer files, special token maps, and configs. If omitted,
synthesized configuration and tokenizer files are extracted directly from the GGUF metadata.
.SH POSITIONAL ARGUMENTS
.TP
.I gguf
Path to the input GGUF model file.
.SH OPTIONS
.SS Output & File Control
.TP
.BI \-o " " OUTPUT ", " \-\-output " " OUTPUT
Output destination directory (default: \fI./<model-name>-NVFP4\fR, or \fI./<model-name>\fR with
\fB\-\-no\-quant\fR and \fBauto\fR).
.TP
.B \-\-overwrite
Overwrite existing files in the output directory without prompting.
.TP
.BI \-\-format " " {hf,modelopt,comfy}
Safetensors layout format. \fBhf\fR (default) produces standard Hugging Face
\fBcompressed-tensors\fR layout (\fI<name>.weight_packed\fR, \fI<name>.weight_scale\fR,
and \fI<name>.weight_global_scale\fR) natively supported by Hugging Face transformers,
vLLM, SGLang, and llama.cpp. \fBmodelopt\fR (or \fBcomfy\fR) produces raw tensors
(\fI<name>.weight\fR, \fI<name>.weight_scale\fR, \fI<name>.weight_scale_2\fR, and \fI<name>.comfy_quant\fR).
.TP
.BI \-\-hf\-template " " DIR
Reference Hugging Face model directory to copy config, tokenizer, and template files from.
.TP
.BI \-\-context\-length " " LEN
Override the maximum sequence/context length recorded in \fIconfig.json\fR.
.TP
.BI \-\-model\-name " " NAME
Override the model name.
.SS Quantization Mode
.TP
.B \-\-no\-quant
Skip learned rounding. Rewrites tensors as float weights.
.TP
.BI \-\-outtype " " {auto,f32,f16,bf16,nvfp4}
Data type for tensors when \fB\-\-no\-quant\fR is used (default: \fBauto\fR).
.TP
.BI \-\-output\-dtype " " {bfloat16,float16}
Data type used for unquantized 2D weights, layer norms, and embeddings (default: \fBbfloat16\fR).
.TP
.B \-\-simple
Round weights directly onto the E2M1 grid without iterative learned rounding.
.SS Optimization & Learning Rate
.TP
.BI \-\-optimizer " " {auto,prodigy,adamw,radam,original}
Optimizer for learned rounding. \fBauto\fR selects Prodigy if installed, otherwise AdamW (default: \fBauto\fR).
.TP
.BI \-\-num\-iter " " N
Maximum optimization iterations per weight matrix (default: 4000).
.TP
.BI \-\-lr " " LR
Initial learning rate (default: 1.0).
.TP
.BI \-\-lr\-patience " " N
Patience steps before decaying the learning rate (default: 5).
.TP
.BI \-\-lr\-min " " MIN
Minimum learning rate floor (default: 1e-8).
.TP
.BI \-\-lr\-schedule " " {plateau,exponential,adaptive}
Learning rate decay schedule (default: \fBplateau\fR).
.TP
.BI \-\-scale\-optimization " " {fixed,iterative,joint}
Scale factor optimization strategy. \fBjoint\fR optimizes block scales simultaneously with codes (default: \fBjoint\fR).
.TP
.BI \-\-scale\-refinement\-rounds " " N
Number of scale refinement passes when using iterative scale optimization (default: 2).
.SS SVD Low-Rank Approximation
.TP
.B \-\-full\-matrix
Perform full SVD instead of randomized low-rank SVD.
.TP
.BI \-\-top\-p " " P
Cumulative energy threshold for singular value truncation (default: 0.2).
.TP
.BI \-\-min\-k " " K
Minimum number of singular vectors retained (default: 128).
.TP
.BI \-\-max\-k " " K
Maximum number of singular vectors retained (default: 1280).
.SS Early Stopping
.TP
.BI \-\-target\-loss " " TARGET
Stop optimization once reconstruction loss reaches this threshold (default: 1e-5). Set 0 to disable.
.TP
.BI \-\-early\-stop\-stall " " STALL
Stop after this many steps without loss improvement (default: 2000).
.TP
.BI \-\-early\-stop\-floor\-stall " " FLOOR
Stop after this many steps without improvement once learning rate reaches its floor (default: 64).
.TP
.BI \-\-early\-stop\-loss " " LOSS
Early stopping absolute loss threshold (default: 5e-9).
.SS Calibration & Bias Correction
.TP
.BI \-\-calib\-samples " " SAMPLES
Number of random normal calibration samples used for bias error compensation (default: 3072).
.TP
.BI \-\-seed " " SEED
Random seed for calibration data generation (default: 42).
.TP
.B \-\-no\-bias\-correction
Leave bias tensors unaltered without applying calibration-based error correction.
.SS Layer Selection
.TP
.B \-\-quantize\-output
Also quantize the final language model head (\fIlm_head.weight\fR). By default, \fIlm_head\fR remains unquantized.
.TP
.BI \-\-exclude " " REGEX ", " \-\-exclude\-layers " " REGEX
Regular expression matching tensor names that should remain unquantized in float format.
.SS Checkpoints & Resuming
.TP
.BI \-\-checkpoint\-dir " " DIR
Directory to store and load intermediate per-tensor quantization checkpoints.
.TP
.B \-\-fresh
Delete existing per-tensor checkpoints before starting.
.TP
.B \-\-keep\-checkpoints
Retain per-tensor checkpoints upon successful completion.
.SS Execution & Environment
.TP
.BI \-\-device " " DEVICE
Compute device: \fBauto\fR, \fBcpu\fR, \fBcuda\fR, or \fBcuda:N\fR (default: \fBauto\fR).
.TP
.B \-\-eager
Load tensors eagerly into memory.
.TP
.B \-\-low\-memory
Stream tensors sequentially to minimize memory footprint.
.TP
.B \-\-heur
Skip layers that convert_to_quant's heuristic rejects.
.TP
.BI \-\-convert\-to\-quant " " PATH
Path to convert_to_quant source checkout if not installed in Python environment.
.TP
.BI \-\-llama\-cpp " " PATH
Path to llama.cpp source checkout (default: \fI~/llama.cpp\fR).
.SS Diagnostics
.TP
.B \-\-verbose
Enable detailed progress and debugging output.
.TP
.B \-\-self\-test
Run GGUF-to-HF unpack and roundtrip self-test and exit.
.TP
.B \-h\fR, \fB\-\-help
Show command-line help and exit.
.SH EXAMPLES
.PP
Convert and quantize a GGUF model to NVFP4 HF format:
.PP
.nf
.RS
gguf_to_nvfp4_hf -o ./meta-llama-3.1-8b-nvfp4 ./meta-llama-3.1-8b.gguf
.RE
.fi
.PP
Convert GGUF to HF using an upstream HF repo as template for tokenizers:
.PP
.nf
.RS
gguf_to_nvfp4_hf --hf-template ./upstream-hf-dir -o ./output-hf-dir ./model.gguf
.RE
.fi
.SH SEE ALSO
.BR hf_to_nvfp4_hf (1),
.BR hf_to_nvfp4_gguf (1),
.BR gguf_to_nvfp4_gguf (1),
.BR safetensors_to_nvfp4_gguf (1)