nvfp4_gguf_quant/man/man1/hf_to_nvfp4_hf.1

.TH HF_TO_NVFP4_HF 1 "November 2026" "nvfp4_gguf_quant" "User Commands"
.SH NAME
hf_to_nvfp4_hf \- quantize Hugging Face models to NVFP4 Hugging Face format
.SH SYNOPSIS
.B hf_to_nvfp4_hf
[\fB\-o\fR \fIOUTPUT\fR]
[\fB\-\-overwrite\fR]
[\fB\-\-format\fR \fIFORMAT\fR]
[\fB\-\-no\-quant\fR]
[\fB\-\-outtype\fR \fITYPE\fR]
[\fB\-\-output\-dtype\fR \fIDTYPE\fR]
[\fB\-\-context\-length\fR \fILEN\fR]
[\fB\-\-model\-name\fR \fINAME\fR]
[\fB\-\-checkpoint\-dir\fR \fIDIR\fR]
[\fB\-\-fresh\fR]
[\fB\-\-keep\-checkpoints\fR]
[\fB\-\-eager\fR]
[\fB\-\-low\-memory\fR]
[\fB\-\-heur\fR]
[\fB\-\-verbose\fR]
[\fB\-\-self\-test\fR]
[\fB\-\-llama\-cpp\fR \fIPATH\fR]
[\fB\-\-convert\-to\-quant\fR \fIPATH\fR]
[\fB\-\-device\fR \fIDEVICE\fR]
[\fB\-\-simple\fR]
[\fB\-\-optimizer\fR \fIOPT\fR]
[\fB\-\-num\-iter\fR \fIN\fR]
[\fB\-\-lr\fR \fILR\fR]
[\fB\-\-lr\-patience\fR \fIN\fR]
[\fB\-\-lr\-min\fR \fIMIN\fR]
[\fB\-\-lr\-schedule\fR \fISCHED\fR]
[\fB\-\-scale\-optimization\fR \fIMODE\fR]
[\fB\-\-scale\-refinement\-rounds\fR \fIN\fR]
[\fB\-\-full\-matrix\fR]
[\fB\-\-top\-p\fR \fIP\fR]
[\fB\-\-min\-k\fR \fIK\fR]
[\fB\-\-max\-k\fR \fIK\fR]
[\fB\-\-early\-stop\-loss\fR \fILOSS\fR]
[\fB\-\-early\-stop\-stall\fR \fISTALL\fR]
[\fB\-\-target\-loss\fR \fITARGET\fR]
[\fB\-\-early\-stop\-floor\-stall\fR \fIFLOOR\fR]
[\fB\-\-calib\-samples\fR \fISAMPLES\fR]
[\fB\-\-seed\fR \fISEED\fR]
[\fB\-\-no\-bias\-correction\fR]
[\fB\-\-quantize\-output\fR]
[\fB\-\-exclude\fR \fIREGEX\fR]
[\fB\-h\fR|\fB\-\-help\fR]
\fImodel\fR
.SH DESCRIPTION
.B hf_to_nvfp4_hf
quantizes a Hugging Face model directory (\fIconfig.json\fR + safetensors) to
NVIDIA FP4 (NVFP4) format packaged into a standard Hugging Face model directory.
.PP
The output directory preserves all non-weight metadata, configuration, tokenizer files,
vocabulary, chat templates, and generation configs. For each quantized weight matrix,
the script writes ModelOpt-compatible packed 4-bit weights alongside per-block E4M3 scale
factors (\fI<name>.weight_scale\fR) and per-tensor scale factors (\fI<name>.weight_scale_2\fR).
.PP
Weights are quantized using learned rounding (via \fBconvert_to_quant\fR) or simple nearest-grid
projection onto the E2M1 floating point grid with two-level scaling. Biases are adjusted
using simulated calibration data unless disabled. Checkpoint-based resumption is supported
per tensor so interrupted runs can resume without recomputing.
.SH POSITIONAL ARGUMENTS
.TP
.I model
Path to a Hugging Face model directory containing \fIconfig.json\fR and one or more \fI.safetensors\fR files.
.SH OPTIONS
.SS Output & File Control
.TP
.BI \-o " " OUTPUT ", " \-\-output " " OUTPUT
Output destination directory (default: \fI./<model-name>-NVFP4\fR, or \fI./<model-name>\fR with
\fB\-\-no\-quant\fR and \fBauto\fR).
.TP
.B \-\-overwrite
Overwrite existing files in the output directory without prompting.
.TP
.BI \-\-format " " {hf,modelopt,comfy}
Safetensors layout format. \fBhf\fR (default) produces standard Hugging Face
\fBcompressed-tensors\fR layout (\fI<name>.weight_packed\fR, \fI<name>.weight_scale\fR,
and \fI<name>.weight_global_scale\fR) natively supported by Hugging Face transformers,
vLLM, SGLang, and llama.cpp. \fBmodelopt\fR (or \fBcomfy\fR) produces raw tensors
(\fI<name>.weight\fR, \fI<name>.weight_scale\fR, \fI<name>.weight_scale_2\fR, and \fI<name>.comfy_quant\fR).
.TP
.BI \-\-context\-length " " LEN
Override the maximum sequence/context length recorded in \fIconfig.json\fR.
.TP
.BI \-\-model\-name " " NAME
Override the model name.
.SS Quantization Mode
.TP
.B \-\-no\-quant
Skip learned rounding. Rewrites or repacks tensors without quantizing.
.TP
.BI \-\-outtype " " {auto,f32,f16,bf16,nvfp4}
Data type for non-quantized weights when learned rounding is skipped (default: \fBauto\fR).
.TP
.BI \-\-output\-dtype " " {bfloat16,float16}
Data type used for unquantized 2D weights, layer norms, and embeddings (default: \fBbfloat16\fR).
.TP
.B \-\-simple
Round weights directly onto the E2M1 grid without iterative learned rounding.
.SS Optimization & Learning Rate
.TP
.BI \-\-optimizer " " {auto,prodigy,adamw,radam,original}
Optimizer for learned rounding. \fBauto\fR selects Prodigy if installed, otherwise AdamW (default: \fBauto\fR).
.TP
.BI \-\-num\-iter " " N
Maximum optimization iterations per weight matrix (default: 4000).
.TP
.BI \-\-lr " " LR
Initial learning rate (default: 1.0).
.TP
.BI \-\-lr\-patience " " N
Patience steps before decaying the learning rate (default: 5).
.TP
.BI \-\-lr\-min " " MIN
Minimum learning rate floor (default: 1e-8).
.TP
.BI \-\-lr\-schedule " " {plateau,exponential,adaptive}
Learning rate decay schedule (default: \fBplateau\fR).
.TP
.BI \-\-scale\-optimization " " {fixed,iterative,joint}
Scale factor optimization strategy. \fBjoint\fR optimizes block scales simultaneously with codes (default: \fBjoint\fR).
.TP
.BI \-\-scale\-refinement\-rounds " " N
Number of scale refinement passes when using iterative scale optimization (default: 2).
.SS SVD Low-Rank Approximation
.TP
.B \-\-full\-matrix
Perform full SVD instead of randomized low-rank SVD.
.TP
.BI \-\-top\-p " " P
Cumulative energy threshold for singular value truncation (default: 0.2).
.TP
.BI \-\-min\-k " " K
Minimum number of singular vectors retained (default: 128).
.TP
.BI \-\-max\-k " " K
Maximum number of singular vectors retained (default: 1280).
.SS Early Stopping
.TP
.BI \-\-target\-loss " " TARGET
Stop optimization once reconstruction loss reaches this threshold (default: 1e-5). Set 0 to disable.
.TP
.BI \-\-early\-stop\-stall " " STALL
Stop after this many steps without loss improvement (default: 2000).
.TP
.BI \-\-early\-stop\-floor\-stall " " FLOOR
Stop after this many steps without improvement once learning rate reaches its floor (default: 64).
.TP
.BI \-\-early\-stop\-loss " " LOSS
Early stopping absolute loss threshold (default: 5e-9).
.SS Calibration & Bias Correction
.TP
.BI \-\-calib\-samples " " SAMPLES
Number of random normal calibration samples used for bias error compensation (default: 3072).
.TP
.BI \-\-seed " " SEED
Random seed for calibration data generation (default: 42).
.TP
.B \-\-no\-bias\-correction
Leave bias tensors unaltered without applying calibration-based error correction.
.SS Layer Selection
.TP
.B \-\-quantize\-output
Also quantize the final language model head (\fIlm_head.weight\fR). By default, \fIlm_head\fR remains unquantized.
.TP
.BI \-\-exclude " " REGEX ", " \-\-exclude\-layers " " REGEX
Regular expression matching tensor names that should remain unquantized in float format.
.SS Checkpoints & Resuming
.TP
.BI \-\-checkpoint\-dir " " DIR
Directory to store and load intermediate per-tensor quantization checkpoints.
.TP
.B \-\-fresh
Delete existing per-tensor checkpoints before starting.
.TP
.B \-\-keep\-checkpoints
Retain per-tensor checkpoints upon successful completion.
.SS Execution & Environment
.TP
.BI \-\-device " " DEVICE
Compute device: \fBauto\fR, \fBcpu\fR, \fBcuda\fR, or \fBcuda:N\fR (default: \fBauto\fR).
.TP
.B \-\-eager
Load tensors eagerly into memory.
.TP
.B \-\-low\-memory
Stream shards sequentially to minimize memory footprint.
.TP
.B \-\-heur
Skip layers that convert_to_quant's heuristic rejects.
.TP
.BI \-\-convert\-to\-quant " " PATH
Path to convert_to_quant source checkout if not installed in Python environment.
.TP
.BI \-\-llama\-cpp " " PATH
Path to llama.cpp source checkout if needed (default: \fI~/llama.cpp\fR).
.SS Diagnostics
.TP
.B \-\-verbose
Enable detailed progress and debugging output.
.TP
.B \-\-self\-test
Run NVFP4 packing and roundtrip self-test and exit.
.TP
.B \-h\fR, \fB\-\-help
Show command-line help and exit.
.SH EXAMPLES
.PP
Quantize a Hugging Face model directory to NVFP4 HF format:
.PP
.nf
.RS
hf_to_nvfp4_hf -o ./meta-llama-3.1-8b-nvfp4 ./meta-llama-3.1-8b
.RE
.fi
.PP
Quantize with fast simple grid rounding without learned iterations:
.PP
.nf
.RS
hf_to_nvfp4_hf --simple -o ./my-model-nvfp4 ./my-model
.RE
.fi
.SH SEE ALSO
.BR hf_to_nvfp4_gguf (1),
.BR gguf_to_nvfp4_hf (1),
.BR hf_to_nvfp4_safetensors (1),
.BR gguf_to_nvfp4_gguf (1)