217 lines
6.9 KiB
Groff
217 lines
6.9 KiB
Groff
.TH GGUF_TO_NVFP4_GGUF 1 "November 2026" "nvfp4_gguf_quant" "User Commands"
|
|
.SH NAME
|
|
gguf_to_nvfp4_gguf \- quantize or repack an existing GGUF model to NVFP4 GGUF
|
|
.SH SYNOPSIS
|
|
.B gguf_to_nvfp4_gguf
|
|
[\fB\-o\fR \fIOUTPUT\fR]
|
|
[\fB\-\-modelfile\-only\fR]
|
|
[\fB\-\-overwrite\fR]
|
|
[\fB\-\-mmproj\fR \fIMMPROJ\fR]
|
|
[\fB\-\-context\-length\fR \fILEN\fR]
|
|
[\fB\-\-no\-quant\fR]
|
|
[\fB\-\-outtype\fR \fITYPE\fR]
|
|
[\fB\-\-llama\-cpp\fR \fIPATH\fR]
|
|
[\fB\-\-convert\-to\-quant\fR \fIPATH\fR]
|
|
[\fB\-\-model\-name\fR \fINAME\fR]
|
|
[\fB\-\-device\fR \fIDEVICE\fR]
|
|
[\fB\-\-simple\fR]
|
|
[\fB\-\-optimizer\fR \fIOPT\fR]
|
|
[\fB\-\-num\-iter\fR \fIN\fR]
|
|
[\fB\-\-lr\fR \fILR\fR]
|
|
[\fB\-\-lr\-patience\fR \fIN\fR]
|
|
[\fB\-\-lr\-min\fR \fIMIN\fR]
|
|
[\fB\-\-lr\-schedule\fR \fISCHED\fR]
|
|
[\fB\-\-scale\-optimization\fR \fIMODE\fR]
|
|
[\fB\-\-scale\-refinement\-rounds\fR \fIN\fR]
|
|
[\fB\-\-full\-matrix\fR]
|
|
[\fB\-\-top\-p\fR \fIP\fR]
|
|
[\fB\-\-min\-k\fR \fIK\fR]
|
|
[\fB\-\-max\-k\fR \fIK\fR]
|
|
[\fB\-\-early\-stop\-loss\fR \fILOSS\fR]
|
|
[\fB\-\-early\-stop\-stall\fR \fISTALL\fR]
|
|
[\fB\-\-target\-loss\fR \fITARGET\fR]
|
|
[\fB\-\-early\-stop\-floor\-stall\fR \fIFLOOR\fR]
|
|
[\fB\-\-calib\-samples\fR \fISAMPLES\fR]
|
|
[\fB\-\-seed\fR \fISEED\fR]
|
|
[\fB\-\-no\-bias\-correction\fR]
|
|
[\fB\-\-quantize\-output\fR]
|
|
[\fB\-\-exclude\fR \fIREGEX\fR]
|
|
[\fB\-\-checkpoint\-dir\fR \fIDIR\fR]
|
|
[\fB\-\-fresh\fR]
|
|
[\fB\-\-keep\-checkpoints\fR]
|
|
[\fB\-\-eager\fR]
|
|
[\fB\-\-no\-temp\-file\fR]
|
|
[\fB\-\-verbose\fR]
|
|
[\fB\-\-self\-test\fR]
|
|
[\fB\-h\fR|\fB\-\-help\fR]
|
|
\fIgguf\fR
|
|
.SH DESCRIPTION
|
|
.B gguf_to_nvfp4_gguf
|
|
reads an existing GGUF model (such as an F16, BF16, or existing quantized checkpoint)
|
|
and converts it to NVIDIA FP4 (NVFP4) format, or repacks an existing NVFP4 GGUF file.
|
|
.PP
|
|
Vocabulary, tokenizer settings, architecture metadata, and tensor names are copied directly
|
|
from the source GGUF header. When quantizing unquantized (float) tensors, learned rounding
|
|
or simple grid rounding is applied using the same optimization flags as \fBhf_to_nvfp4_gguf\fR.
|
|
.PP
|
|
An accompanying Ollama \fBModelfile\fR is written alongside the output GGUF, configured with
|
|
the source model's parameters and templates.
|
|
.SH POSITIONAL ARGUMENTS
|
|
.TP
|
|
.I gguf
|
|
Input GGUF file path.
|
|
.SH OPTIONS
|
|
.SS Output & File Control
|
|
.TP
|
|
.BI \-o " " OUTPUT ", " \-\-output " " OUTPUT
|
|
Output directory (default: \fI./<name>-NVFP4\fR, or \fI./<name>\fR with \fB\-\-no\-quant\fR and \fBauto\fR).
|
|
If ending in \fI.gguf\fR, specifies the destination file directly.
|
|
.TP
|
|
.B \-\-overwrite
|
|
Overwrite existing output GGUF, Modelfile, or copied projector without prompting.
|
|
.TP
|
|
.B \-\-modelfile\-only
|
|
Write the Ollama Modelfile and skip quantization and GGUF generation.
|
|
.TP
|
|
.BI \-\-mmproj " " MMPROJ
|
|
Path to a vision projector GGUF to bundle into the output directory and add as another \fBFROM\fR layer.
|
|
May be repeated for multi-modal models.
|
|
.TP
|
|
.BI \-\-context\-length " " LEN
|
|
Override maximum context length stored in the GGUF metadata.
|
|
.TP
|
|
.BI \-\-model\-name " " NAME
|
|
Override the model name stored in the GGUF metadata.
|
|
.SS Quantization Mode
|
|
.TP
|
|
.B \-\-no\-quant
|
|
Skip learned rounding. Writes a float GGUF, or copies/repacks an existing NVFP4 GGUF when combined with \fB\-\-outtype nvfp4\fR.
|
|
.TP
|
|
.BI \-\-outtype " " {auto,f32,f16,bf16,nvfp4}
|
|
Target data type when learned rounding is skipped. \fBnvfp4\fR copies an already NVFP4 GGUF without re-optimizing;
|
|
\fBauto\fR leaves tensors as stored in the source file (default: \fBauto\fR).
|
|
.TP
|
|
.B \-\-simple
|
|
Round each value directly onto the E2M1 grid without learned rounding (fast, non-iterative).
|
|
.TP
|
|
.B \-\-quantize\-output
|
|
Also quantize the output (\fIlm_head\fR) matrix. By default, output projection layers are left unquantized.
|
|
.TP
|
|
.BI \-\-exclude " " REGEX
|
|
Regular expression matching GGUF tensor names that should remain in their source precision.
|
|
.SS Optimization & Learned Rounding
|
|
.TP
|
|
.BI \-\-optimizer " " {auto,prodigy,adamw,radam,original}
|
|
Learned-rounding optimization algorithm (default: \fBauto\fR; selects Prodigy if available, else AdamW).
|
|
.TP
|
|
.BI \-\-num\-iter " " N
|
|
Learned-rounding optimization iterations per matrix (default: 4000).
|
|
.TP
|
|
.BI \-\-lr " " LR
|
|
Initial learning rate (default: 1.0).
|
|
.TP
|
|
.BI \-\-lr\-patience " " N
|
|
Patience steps before decaying the learning rate (default: 5).
|
|
.TP
|
|
.BI \-\-lr\-min " " MIN
|
|
Lowest learning rate floor (default: 1e-8).
|
|
.TP
|
|
.BI \-\-lr\-schedule " " {plateau,exponential,adaptive}
|
|
Learning rate decay schedule (default: \fBplateau\fR).
|
|
.TP
|
|
.BI \-\-scale\-optimization " " {fixed,iterative,joint}
|
|
Block scale tuning strategy: \fBfixed\fR, \fBiterative\fR, or \fBjoint\fR (default: \fBjoint\fR).
|
|
.TP
|
|
.BI \-\-scale\-refinement\-rounds " " N
|
|
Scale refresh iterations when using \fBiterative\fR scale optimization (default: 2).
|
|
.TP
|
|
.B \-\-full\-matrix
|
|
Use full SVD instead of low-rank SVD for reconstruction objective.
|
|
.TP
|
|
.BI \-\-top\-p " " P ", " \-\-min\-k " " K ", " \-\-max\-k " " K
|
|
Singular value decomposition rank selection bounds.
|
|
.TP
|
|
.BI \-\-target\-loss " " TARGET
|
|
Target reconstruction loss threshold for early exit (default: 1e-5; 0 disables).
|
|
.TP
|
|
.BI \-\-early\-stop\-stall " " STALL
|
|
Early stop threshold in steps without loss reduction (default: 2000).
|
|
.TP
|
|
.BI \-\-early\-stop\-floor\-stall " " FLOOR
|
|
Steps without improvement after reaching \fB\-\-lr\-min\fR before terminating matrix search (default: 64; 0 disables).
|
|
.TP
|
|
.BI \-\-calib\-samples " " SAMPLES
|
|
Random calibration vectors used for sibling bias correction (default: 3072).
|
|
.TP
|
|
.BI \-\-seed " " SEED
|
|
Random seed (default: 42).
|
|
.TP
|
|
.B \-\-no\-bias\-correction
|
|
Disable sibling bias correction.
|
|
.SS System & Execution
|
|
.TP
|
|
.BI \-\-device " " DEVICE
|
|
Compute device: \fBauto\fR, \fBcpu\fR, \fBcuda\fR, or \fBcuda:N\fR (default: \fBauto\fR).
|
|
.TP
|
|
.BI \-\-llama\-cpp " " PATH
|
|
Path to local llama.cpp repository checkout (default: \fI~/llama.cpp\fR).
|
|
.TP
|
|
.BI \-\-convert\-to\-quant " " PATH
|
|
Path to local convert_to_quant repository checkout.
|
|
.TP
|
|
.BI \-\-checkpoint\-dir " " DIR
|
|
Directory for per-tensor resume checkpoints.
|
|
.TP
|
|
.B \-\-fresh
|
|
Delete checkpoint directory before starting.
|
|
.TP
|
|
.B \-\-keep\-checkpoints
|
|
Keep per-tensor checkpoints on disk after completion.
|
|
.TP
|
|
.B \-\-eager
|
|
Map and load tensors up front.
|
|
.TP
|
|
.B \-\-no\-temp\-file
|
|
Keep finished GGUF tensors in RAM rather than a temporary file.
|
|
.TP
|
|
.B \-\-verbose
|
|
Enable verbose logging.
|
|
.TP
|
|
.B \-\-self\-test
|
|
Verify NVFP4 packing against llama.cpp dequantizer and exit.
|
|
.SH EXAMPLES
|
|
.TP
|
|
.B Quantize an F16 GGUF to NVFP4:
|
|
.RS
|
|
.nf
|
|
gguf_to_nvfp4_gguf -o ./llama3-nvfp4 ./llama-3-8b-f16.gguf
|
|
.fi
|
|
.RE
|
|
.TP
|
|
.B Fast conversion with simple grid rounding:
|
|
.RS
|
|
.nf
|
|
gguf_to_nvfp4_gguf --simple -o ./qwen-nvfp4 ./qwen2.5-7b-bf16.gguf
|
|
.fi
|
|
.RE
|
|
.TP
|
|
.B Repack existing NVFP4 GGUF into a new directory with Modelfile:
|
|
.RS
|
|
.nf
|
|
gguf_to_nvfp4_gguf --no-quant --outtype nvfp4 -o ./repacked ./model-nvfp4.gguf
|
|
.fi
|
|
.RE
|
|
.SH EXIT STATUS
|
|
.TP
|
|
.B 0
|
|
Success.
|
|
.TP
|
|
.B 1
|
|
An error occurred during quantization or repacking.
|
|
.SH SEE ALSO
|
|
.BR hf_to_nvfp4_gguf (1),
|
|
.BR safetensors_to_nvfp4_gguf (1),
|
|
.BR gguf_info (1),
|
|
.BR ollama_nvfp4 (1)
|
|
|