242 lines
8.2 KiB
Groff
242 lines
8.2 KiB
Groff
.TH GGUF_TO_NVFP4_HF 1 "November 2026" "nvfp4_gguf_quant" "User Commands"
|
|
.SH NAME
|
|
gguf_to_nvfp4_hf \- convert and quantize GGUF models to NVFP4 Hugging Face format
|
|
.SH SYNOPSIS
|
|
.B gguf_to_nvfp4_hf
|
|
[\fB\-o\fR \fIOUTPUT\fR]
|
|
[\fB\-\-overwrite\fR]
|
|
[\fB\-\-format\fR \fIFORMAT\fR]
|
|
[\fB\-\-no\-quant\fR]
|
|
[\fB\-\-outtype\fR \fITYPE\fR]
|
|
[\fB\-\-output\-dtype\fR \fIDTYPE\fR]
|
|
[\fB\-\-hf\-template\fR \fIDIR\fR]
|
|
[\fB\-\-context\-length\fR \fILEN\fR]
|
|
[\fB\-\-model\-name\fR \fINAME\fR]
|
|
[\fB\-\-checkpoint\-dir\fR \fIDIR\fR]
|
|
[\fB\-\-fresh\fR]
|
|
[\fB\-\-keep\-checkpoints\fR]
|
|
[\fB\-\-eager\fR]
|
|
[\fB\-\-low\-memory\fR]
|
|
[\fB\-\-heur\fR]
|
|
[\fB\-\-verbose\fR]
|
|
[\fB\-\-self\-test\fR]
|
|
[\fB\-\-llama\-cpp\fR \fIPATH\fR]
|
|
[\fB\-\-convert\-to\-quant\fR \fIPATH\fR]
|
|
[\fB\-\-device\fR \fIDEVICE\fR]
|
|
[\fB\-\-simple\fR]
|
|
[\fB\-\-optimizer\fR \fIOPT\fR]
|
|
[\fB\-\-num\-iter\fR \fIN\fR]
|
|
[\fB\-\-lr\fR \fILR\fR]
|
|
[\fB\-\-lr\-patience\fR \fIN\fR]
|
|
[\fB\-\-lr\-min\fR \fIMIN\fR]
|
|
[\fB\-\-lr\-schedule\fR \fISCHED\fR]
|
|
[\fB\-\-scale\-optimization\fR \fIMODE\fR]
|
|
[\fB\-\-scale\-refinement\-rounds\fR \fIN\fR]
|
|
[\fB\-\-full\-matrix\fR]
|
|
[\fB\-\-top\-p\fR \fIP\fR]
|
|
[\fB\-\-min\-k\fR \fIK\fR]
|
|
[\fB\-\-max\-k\fR \fIK\fR]
|
|
[\fB\-\-early\-stop\-loss\fR \fILOSS\fR]
|
|
[\fB\-\-early\-stop\-stall\fR \fISTALL\fR]
|
|
[\fB\-\-target\-loss\fR \fITARGET\fR]
|
|
[\fB\-\-early\-stop\-floor\-stall\fR \fIFLOOR\fR]
|
|
[\fB\-\-calib\-samples\fR \fISAMPLES\fR]
|
|
[\fB\-\-seed\fR \fISEED\fR]
|
|
[\fB\-\-no\-bias\-correction\fR]
|
|
[\fB\-\-quantize\-output\fR]
|
|
[\fB\-\-exclude\fR \fIREGEX\fR]
|
|
[\fB\-h\fR|\fB\-\-help\fR]
|
|
\fIgguf\fR
|
|
.SH DESCRIPTION
|
|
.B gguf_to_nvfp4_hf
|
|
converts a GGUF model file into a standard Hugging Face model directory
|
|
(\fIconfig.json\fR + safetensors) with NVIDIA FP4 (NVFP4) quantized weights.
|
|
.PP
|
|
GGUF tensor names are reverse-mapped to canonical Hugging Face conventions
|
|
(e.g., \fIblk.0.attn_q.weight\fR maps to \fImodel.layers.0.self_attn.q_proj.weight\fR),
|
|
and RoPE permutations for Q and K are inverted.
|
|
.PP
|
|
If the source GGUF contains float or non-NVFP4 weights, 2D weight matrices are quantized
|
|
using convert_to_quant's learned rounding (or simple round-to-grid with \fB\-\-simple\fR).
|
|
If the source GGUF already contains NVFP4 super-blocks, they are unpacked and repacked
|
|
into ModelOpt-compatible safetensors without loss of precision.
|
|
.PP
|
|
If \fB\-\-no\-quant\fR is specified, the model is dequantized to standard float format
|
|
(\fI--outtype f32, f16, bf16, or auto\fR).
|
|
.PP
|
|
An optional reference Hugging Face model directory can be provided with \fB\-\-hf\-template\fR
|
|
to copy exact upstream tokenizer files, special token maps, and configs. If omitted,
|
|
synthesized configuration and tokenizer files are extracted directly from the GGUF metadata.
|
|
.SH POSITIONAL ARGUMENTS
|
|
.TP
|
|
.I gguf
|
|
Path to the input GGUF model file.
|
|
.SH OPTIONS
|
|
.SS Output & File Control
|
|
.TP
|
|
.BI \-o " " OUTPUT ", " \-\-output " " OUTPUT
|
|
Output destination directory (default: \fI./<model-name>-NVFP4\fR, or \fI./<model-name>\fR with
|
|
\fB\-\-no\-quant\fR and \fBauto\fR).
|
|
.TP
|
|
.B \-\-overwrite
|
|
Overwrite existing files in the output directory without prompting.
|
|
.TP
|
|
.BI \-\-format " " {hf,modelopt,comfy}
|
|
Safetensors layout format. \fBhf\fR (default) produces standard Hugging Face
|
|
\fBcompressed-tensors\fR layout (\fI<name>.weight_packed\fR, \fI<name>.weight_scale\fR,
|
|
and \fI<name>.weight_global_scale\fR) natively supported by Hugging Face transformers,
|
|
vLLM, SGLang, and llama.cpp. \fBmodelopt\fR (or \fBcomfy\fR) produces raw tensors
|
|
(\fI<name>.weight\fR, \fI<name>.weight_scale\fR, \fI<name>.weight_scale_2\fR, and \fI<name>.comfy_quant\fR).
|
|
.TP
|
|
.BI \-\-hf\-template " " DIR
|
|
Reference Hugging Face model directory to copy config, tokenizer, and template files from.
|
|
.TP
|
|
.BI \-\-context\-length " " LEN
|
|
Override the maximum sequence/context length recorded in \fIconfig.json\fR.
|
|
.TP
|
|
.BI \-\-model\-name " " NAME
|
|
Override the model name.
|
|
.SS Quantization Mode
|
|
.TP
|
|
.B \-\-no\-quant
|
|
Skip learned rounding. Rewrites tensors as float weights.
|
|
.TP
|
|
.BI \-\-outtype " " {auto,f32,f16,bf16,nvfp4}
|
|
Data type for tensors when \fB\-\-no\-quant\fR is used (default: \fBauto\fR).
|
|
.TP
|
|
.BI \-\-output\-dtype " " {bfloat16,float16}
|
|
Data type used for unquantized 2D weights, layer norms, and embeddings (default: \fBbfloat16\fR).
|
|
.TP
|
|
.B \-\-simple
|
|
Round weights directly onto the E2M1 grid without iterative learned rounding.
|
|
.SS Optimization & Learning Rate
|
|
.TP
|
|
.BI \-\-optimizer " " {auto,prodigy,adamw,radam,original}
|
|
Optimizer for learned rounding. \fBauto\fR selects Prodigy if installed, otherwise AdamW (default: \fBauto\fR).
|
|
.TP
|
|
.BI \-\-num\-iter " " N
|
|
Maximum optimization iterations per weight matrix (default: 4000).
|
|
.TP
|
|
.BI \-\-lr " " LR
|
|
Initial learning rate (default: 1.0).
|
|
.TP
|
|
.BI \-\-lr\-patience " " N
|
|
Patience steps before decaying the learning rate (default: 5).
|
|
.TP
|
|
.BI \-\-lr\-min " " MIN
|
|
Minimum learning rate floor (default: 1e-8).
|
|
.TP
|
|
.BI \-\-lr\-schedule " " {plateau,exponential,adaptive}
|
|
Learning rate decay schedule (default: \fBplateau\fR).
|
|
.TP
|
|
.BI \-\-scale\-optimization " " {fixed,iterative,joint}
|
|
Scale factor optimization strategy. \fBjoint\fR optimizes block scales simultaneously with codes (default: \fBjoint\fR).
|
|
.TP
|
|
.BI \-\-scale\-refinement\-rounds " " N
|
|
Number of scale refinement passes when using iterative scale optimization (default: 2).
|
|
.SS SVD Low-Rank Approximation
|
|
.TP
|
|
.B \-\-full\-matrix
|
|
Perform full SVD instead of randomized low-rank SVD.
|
|
.TP
|
|
.BI \-\-top\-p " " P
|
|
Cumulative energy threshold for singular value truncation (default: 0.2).
|
|
.TP
|
|
.BI \-\-min\-k " " K
|
|
Minimum number of singular vectors retained (default: 128).
|
|
.TP
|
|
.BI \-\-max\-k " " K
|
|
Maximum number of singular vectors retained (default: 1280).
|
|
.SS Early Stopping
|
|
.TP
|
|
.BI \-\-target\-loss " " TARGET
|
|
Stop optimization once reconstruction loss reaches this threshold (default: 1e-5). Set 0 to disable.
|
|
.TP
|
|
.BI \-\-early\-stop\-stall " " STALL
|
|
Stop after this many steps without loss improvement (default: 2000).
|
|
.TP
|
|
.BI \-\-early\-stop\-floor\-stall " " FLOOR
|
|
Stop after this many steps without improvement once learning rate reaches its floor (default: 64).
|
|
.TP
|
|
.BI \-\-early\-stop\-loss " " LOSS
|
|
Early stopping absolute loss threshold (default: 5e-9).
|
|
.SS Calibration & Bias Correction
|
|
.TP
|
|
.BI \-\-calib\-samples " " SAMPLES
|
|
Number of random normal calibration samples used for bias error compensation (default: 3072).
|
|
.TP
|
|
.BI \-\-seed " " SEED
|
|
Random seed for calibration data generation (default: 42).
|
|
.TP
|
|
.B \-\-no\-bias\-correction
|
|
Leave bias tensors unaltered without applying calibration-based error correction.
|
|
.SS Layer Selection
|
|
.TP
|
|
.B \-\-quantize\-output
|
|
Also quantize the final language model head (\fIlm_head.weight\fR). By default, \fIlm_head\fR remains unquantized.
|
|
.TP
|
|
.BI \-\-exclude " " REGEX ", " \-\-exclude\-layers " " REGEX
|
|
Regular expression matching tensor names that should remain unquantized in float format.
|
|
.SS Checkpoints & Resuming
|
|
.TP
|
|
.BI \-\-checkpoint\-dir " " DIR
|
|
Directory to store and load intermediate per-tensor quantization checkpoints.
|
|
.TP
|
|
.B \-\-fresh
|
|
Delete existing per-tensor checkpoints before starting.
|
|
.TP
|
|
.B \-\-keep\-checkpoints
|
|
Retain per-tensor checkpoints upon successful completion.
|
|
.SS Execution & Environment
|
|
.TP
|
|
.BI \-\-device " " DEVICE
|
|
Compute device: \fBauto\fR, \fBcpu\fR, \fBcuda\fR, or \fBcuda:N\fR (default: \fBauto\fR).
|
|
.TP
|
|
.B \-\-eager
|
|
Load tensors eagerly into memory.
|
|
.TP
|
|
.B \-\-low\-memory
|
|
Stream tensors sequentially to minimize memory footprint.
|
|
.TP
|
|
.B \-\-heur
|
|
Skip layers that convert_to_quant's heuristic rejects.
|
|
.TP
|
|
.BI \-\-convert\-to\-quant " " PATH
|
|
Path to convert_to_quant source checkout if not installed in Python environment.
|
|
.TP
|
|
.BI \-\-llama\-cpp " " PATH
|
|
Path to llama.cpp source checkout (default: \fI~/llama.cpp\fR).
|
|
.SS Diagnostics
|
|
.TP
|
|
.B \-\-verbose
|
|
Enable detailed progress and debugging output.
|
|
.TP
|
|
.B \-\-self\-test
|
|
Run GGUF-to-HF unpack and roundtrip self-test and exit.
|
|
.TP
|
|
.B \-h\fR, \fB\-\-help
|
|
Show command-line help and exit.
|
|
.SH EXAMPLES
|
|
.PP
|
|
Convert and quantize a GGUF model to NVFP4 HF format:
|
|
.PP
|
|
.nf
|
|
.RS
|
|
gguf_to_nvfp4_hf -o ./meta-llama-3.1-8b-nvfp4 ./meta-llama-3.1-8b.gguf
|
|
.RE
|
|
.fi
|
|
.PP
|
|
Convert GGUF to HF using an upstream HF repo as template for tokenizers:
|
|
.PP
|
|
.nf
|
|
.RS
|
|
gguf_to_nvfp4_hf --hf-template ./upstream-hf-dir -o ./output-hf-dir ./model.gguf
|
|
.RE
|
|
.fi
|
|
.SH SEE ALSO
|
|
.BR hf_to_nvfp4_hf (1),
|
|
.BR hf_to_nvfp4_gguf (1),
|
|
.BR gguf_to_nvfp4_gguf (1),
|
|
.BR safetensors_to_nvfp4_gguf (1)
|
|
|