170 lines
5.4 KiB
Groff
170 lines
5.4 KiB
Groff
.TH HF_TO_NVFP4_SAFETENSORS 1 "November 2026" "nvfp4_gguf_quant" "User Commands"
|
|
.SH NAME
|
|
hf_to_nvfp4_safetensors \- quantize a Hugging Face directory to a ComfyUI NVFP4 safetensors file
|
|
.SH SYNOPSIS
|
|
.B hf_to_nvfp4_safetensors
|
|
[\fB\-o\fR \fIOUTPUT\fR]
|
|
[\fB\-\-overwrite\fR]
|
|
[\fB\-\-output\-dtype\fR \fIDTYPE\fR]
|
|
[\fB\-\-low\-memory\fR]
|
|
[\fB\-\-heur\fR]
|
|
[\fB\-\-exclude\-layers\fR \fIREGEX\fR]
|
|
[\fB\-\-convert\-to\-quant\fR \fIPATH\fR]
|
|
[\fB\-\-simple\fR]
|
|
[\fB\-\-optimizer\fR \fIOPT\fR]
|
|
[\fB\-\-num\-iter\fR \fIN\fR]
|
|
[\fB\-\-lr\fR \fILR\fR]
|
|
[\fB\-\-lr\-min\fR \fIMIN\fR]
|
|
[\fB\-\-lr\-patience\fR \fIN\fR]
|
|
[\fB\-\-lr\-schedule\fR \fISCHED\fR]
|
|
[\fB\-\-scale\-optimization\fR \fIMODE\fR]
|
|
[\fB\-\-scale\-refinement\-rounds\fR \fIN\fR]
|
|
[\fB\-\-full\-matrix\fR]
|
|
[\fB\-\-top\-p\fR \fIP\fR]
|
|
[\fB\-\-min\-k\fR \fIK\fR]
|
|
[\fB\-\-max\-k\fR \fIK\fR]
|
|
[\fB\-\-early\-stop\-loss\fR \fILOSS\fR]
|
|
[\fB\-\-early\-stop\-stall\fR \fISTALL\fR]
|
|
[\fB\-\-calib\-samples\fR \fISAMPLES\fR]
|
|
[\fB\-\-seed\fR \fISEED\fR]
|
|
[\fB\-\-use\-speed\fR]
|
|
[\fB\-\-verbose\fR]
|
|
[\fB\-h\fR|\fB\-\-help\fR]
|
|
\fImodel\fR
|
|
.SH DESCRIPTION
|
|
.B hf_to_nvfp4_safetensors
|
|
quantizes a Hugging Face model directory (\fIconfig.json\fR and sharded \fI.safetensors\fR files)
|
|
into a single consolidated \fI.safetensors\fR file formatted for ComfyUI's native NVFP4 loader.
|
|
.PP
|
|
For each quantized weight matrix, the script outputs packed NVFP4 bytes alongside
|
|
companion scaling tensors:
|
|
.IP \(bu 2
|
|
\fI<name>.weight\fR (packed NVFP4 data, two 4-bit values per byte)
|
|
.IP \(bu 2
|
|
\fI<name>.weight_scale\fR (per-block E4M3 scale factors)
|
|
.IP \(bu 2
|
|
\fI<name>.weight_scale_2\fR (per-tensor scale factor)
|
|
.IP \(bu 2
|
|
\fI<name>.comfy_quant\fR (metadata tensor specifying quantization type)
|
|
.PP
|
|
A unified \fI_quantization_metadata\fR header is embedded in the output safetensors header.
|
|
When the source model is split across multiple shards, each shard is quantized in a temporary
|
|
workspace and then efficiently merged into the final single output file.
|
|
.PP
|
|
Normalization layers, embeddings, biases, and layers matching \fB\-\-exclude\-layers\fR
|
|
remain in the target float format (controlled by \fB\-\-output\-dtype\fR).
|
|
.SH POSITIONAL ARGUMENTS
|
|
.TP
|
|
.I model
|
|
Hugging Face model directory containing \fIconfig.json\fR and \fI.safetensors\fR weights.
|
|
.SH OPTIONS
|
|
.SS Output & Memory Control
|
|
.TP
|
|
.BI \-o " " OUTPUT ", " \-\-output " " OUTPUT
|
|
Output \fI.safetensors\fR file path (default: \fI./<model-name>-NVFP4.safetensors\fR).
|
|
.TP
|
|
.B \-\-overwrite
|
|
Overwrite an existing output file without prompting.
|
|
.TP
|
|
.BI \-\-output\-dtype " " {bfloat16,float16}
|
|
Data type recorded for quantized layer scales and used for unquantized 2D weights (default: \fBbfloat16\fR).
|
|
.TP
|
|
.B \-\-low\-memory
|
|
Stream each shard sequentially through quantization rather than loading the entire model shard into RAM at once.
|
|
.TP
|
|
.B \-\-heur
|
|
Skip layers that convert_to_quant's built-in size heuristic rejects.
|
|
.TP
|
|
.BI \-\-exclude\-layers " " REGEX
|
|
Regular expression matching layer or tensor names to leave unquantized in float format.
|
|
.SS Quantization & Optimization
|
|
.TP
|
|
.B \-\-simple
|
|
Round onto the E2M1 grid directly without iterative learned rounding.
|
|
.TP
|
|
.BI \-\-optimizer " " {auto,prodigy,adamw,radam,original}
|
|
Learned-rounding optimizer. \fBauto\fR uses Prodigy when installed, otherwise AdamW (default: \fBauto\fR).
|
|
.TP
|
|
.BI \-\-num\-iter " " N
|
|
Learned-rounding iterations per matrix (default: 4000).
|
|
.TP
|
|
.BI \-\-lr " " LR
|
|
Initial learning rate (default: 1.0).
|
|
.TP
|
|
.BI \-\-lr\-min " " MIN
|
|
Lowest learning rate floor (default: 1e-8).
|
|
.TP
|
|
.BI \-\-lr\-patience " " N
|
|
Patience steps before decaying learning rate (default: 5).
|
|
.TP
|
|
.BI \-\-lr\-schedule " " {plateau,exponential,adaptive}
|
|
Learning rate decay schedule (default: \fBplateau\fR).
|
|
.TP
|
|
.BI \-\-scale\-optimization " " {fixed,iterative,joint}
|
|
Scale tuning strategy: \fBfixed\fR, \fBiterative\fR, or \fBjoint\fR (default: \fBjoint\fR).
|
|
.TP
|
|
.BI \-\-scale\-refinement\-rounds " " N
|
|
Scale refresh iterations when using iterative scale optimization (default: 2).
|
|
.TP
|
|
.B \-\-full\-matrix
|
|
Use full SVD instead of low-rank SVD.
|
|
.TP
|
|
.BI \-\-top\-p " " P ", " \-\-min\-k " " K ", " \-\-max\-k " " K
|
|
Singular value decomposition rank selection bounds.
|
|
.TP
|
|
.BI \-\-early\-stop\-loss " " LOSS
|
|
Early stopping loss target (default: 5e-9).
|
|
.TP
|
|
.BI \-\-early\-stop\-stall " " STALL
|
|
Stop after this many steps without loss improvement (default: 2000).
|
|
.TP
|
|
.BI \-\-calib\-samples " " SAMPLES
|
|
Random Gaussian calibration inputs for bias correction (default: 3072).
|
|
.TP
|
|
.BI \-\-seed " " SEED
|
|
Random seed (default: 42).
|
|
.TP
|
|
.B \-\-use\-speed
|
|
Enable Prodigy optimizer's \fIuse_speed\fR mode.
|
|
.SS System & Execution
|
|
.TP
|
|
.BI \-\-convert\-to\-quant " " PATH
|
|
Path to local convert_to_quant repository checkout.
|
|
.TP
|
|
.B \-\-verbose
|
|
Enable verbose output during shard quantization and merge.
|
|
.SH EXAMPLES
|
|
.TP
|
|
.B Quantize a model directory into a ComfyUI NVFP4 safetensors file:
|
|
.RS
|
|
.nf
|
|
hf_to_nvfp4_safetensors -o gemma-4-12B-it-NVFP4.safetensors ./gemma-4-12B-it
|
|
.fi
|
|
.RE
|
|
.TP
|
|
.B Low-memory streaming mode with float16 weights:
|
|
.RS
|
|
.nf
|
|
hf_to_nvfp4_safetensors --low-memory --output-dtype float16 -o ./model-nvfp4.safetensors ./model
|
|
.fi
|
|
.RE
|
|
.TP
|
|
.B Exclude specific projection layers:
|
|
.RS
|
|
.nf
|
|
hf_to_nvfp4_safetensors --exclude-layers ".*gate_proj.*" -o ./model-nvfp4.safetensors ./model
|
|
.fi
|
|
.RE
|
|
.SH EXIT STATUS
|
|
.TP
|
|
.B 0
|
|
Success.
|
|
.TP
|
|
.B 1
|
|
An error occurred during quantization or shard merging.
|
|
.SH SEE ALSO
|
|
.BR safetensors_to_nvfp4_gguf (1),
|
|
.BR hf_to_nvfp4_gguf (1),
|
|
.BR hf_info (1)
|
|
|