.TH GGUF_TO_NVFP4_GGUF 1 "November 2026" "nvfp4_gguf_quant" "User Commands"
.SH NAME
gguf_to_nvfp4_gguf \- quantize or repack an existing GGUF model to NVFP4 GGUF
.SH SYNOPSIS
.B gguf_to_nvfp4_gguf
[\fB\-o\fR \fIOUTPUT\fR]
[\fB\-\-modelfile\-only\fR]
[\fB\-\-overwrite\fR]
[\fB\-\-mmproj\fR \fIMMPROJ\fR]
[\fB\-\-context\-length\fR \fILEN\fR]
[\fB\-\-no\-quant\fR]
[\fB\-\-outtype\fR \fITYPE\fR]
[\fB\-\-llama\-cpp\fR \fIPATH\fR]
[\fB\-\-convert\-to\-quant\fR \fIPATH\fR]
[\fB\-\-model\-name\fR \fINAME\fR]
[\fB\-\-device\fR \fIDEVICE\fR]
[\fB\-\-simple\fR]
[\fB\-\-optimizer\fR \fIOPT\fR]
[\fB\-\-num\-iter\fR \fIN\fR]
[\fB\-\-lr\fR \fILR\fR]
[\fB\-\-lr\-patience\fR \fIN\fR]
[\fB\-\-lr\-min\fR \fIMIN\fR]
[\fB\-\-lr\-schedule\fR \fISCHED\fR]
[\fB\-\-scale\-optimization\fR \fIMODE\fR]
[\fB\-\-scale\-refinement\-rounds\fR \fIN\fR]
[\fB\-\-full\-matrix\fR]
[\fB\-\-top\-p\fR \fIP\fR]
[\fB\-\-min\-k\fR \fIK\fR]
[\fB\-\-max\-k\fR \fIK\fR]
[\fB\-\-early\-stop\-loss\fR \fILOSS\fR]
[\fB\-\-early\-stop\-stall\fR \fISTALL\fR]
[\fB\-\-target\-loss\fR \fITARGET\fR]
[\fB\-\-early\-stop\-floor\-stall\fR \fIFLOOR\fR]
[\fB\-\-calib\-samples\fR \fISAMPLES\fR]
[\fB\-\-seed\fR \fISEED\fR]
[\fB\-\-no\-bias\-correction\fR]
[\fB\-\-quantize\-output\fR]
[\fB\-\-exclude\fR \fIREGEX\fR]
[\fB\-\-checkpoint\-dir\fR \fIDIR\fR]
[\fB\-\-fresh\fR]
[\fB\-\-keep\-checkpoints\fR]
[\fB\-\-eager\fR]
[\fB\-\-no\-temp\-file\fR]
[\fB\-\-verbose\fR]
[\fB\-\-self\-test\fR]
[\fB\-h\fR|\fB\-\-help\fR]
\fIgguf\fR
.SH DESCRIPTION
.B gguf_to_nvfp4_gguf
reads an existing GGUF model (such as an F16, BF16, or existing quantized checkpoint)
and converts it to NVIDIA FP4 (NVFP4) format, or repacks an existing NVFP4 GGUF file.
.PP
Vocabulary, tokenizer settings, architecture metadata, and tensor names are copied directly
from the source GGUF header. When quantizing unquantized (float) tensors, learned rounding
or simple grid rounding is applied using the same optimization flags as \fBhf_to_nvfp4_gguf\fR.
.PP
An accompanying Ollama \fBModelfile\fR is written alongside the output GGUF, configured with
the source model's parameters and templates.
.SH POSITIONAL ARGUMENTS
.TP
.I gguf
Input GGUF file path.
.SH OPTIONS
.SS Output & File Control
.TP
.BI \-o " " OUTPUT ", " \-\-output " " OUTPUT
Output directory (default: \fI./<name>-NVFP4\fR, or \fI./<name>\fR with \fB\-\-no\-quant\fR and \fBauto\fR).
If ending in \fI.gguf\fR, specifies the destination file directly.
.TP
.B \-\-overwrite
Overwrite existing output GGUF, Modelfile, or copied projector without prompting.
.TP
.B \-\-modelfile\-only
Write the Ollama Modelfile and skip quantization and GGUF generation.
.TP
.BI \-\-mmproj " " MMPROJ
Path to a vision projector GGUF to bundle into the output directory and add as another \fBFROM\fR layer.
May be repeated for multi-modal models.
.TP
.BI \-\-context\-length " " LEN
Override maximum context length stored in the GGUF metadata.
.TP
.BI \-\-model\-name " " NAME
Override the model name stored in the GGUF metadata.
.SS Quantization Mode
.TP
.B \-\-no\-quant
Skip learned rounding. Writes a float GGUF, or copies/repacks an existing NVFP4 GGUF when combined with \fB\-\-outtype nvfp4\fR.
.TP
.BI \-\-outtype " " {auto,f32,f16,bf16,nvfp4}
Target data type when learned rounding is skipped. \fBnvfp4\fR copies an already NVFP4 GGUF without re-optimizing;
\fBauto\fR leaves tensors as stored in the source file (default: \fBauto\fR).
.TP
.B \-\-simple
Round each value directly onto the E2M1 grid without learned rounding (fast, non-iterative).
.TP
.B \-\-quantize\-output
Also quantize the output (\fIlm_head\fR) matrix. By default, output projection layers are left unquantized.
.TP
.BI \-\-exclude " " REGEX
Regular expression matching GGUF tensor names that should remain in their source precision.
.SS Optimization & Learned Rounding
.TP
.BI \-\-optimizer " " {auto,prodigy,adamw,radam,original}
Learned-rounding optimization algorithm (default: \fBauto\fR; selects Prodigy if available, else AdamW).
.TP
.BI \-\-num\-iter " " N
Learned-rounding optimization iterations per matrix (default: 4000).
.TP
.BI \-\-lr " " LR
Initial learning rate (default: 1.0).
.TP
.BI \-\-lr\-patience " " N
Patience steps before decaying the learning rate (default: 5).
.TP
.BI \-\-lr\-min " " MIN
Lowest learning rate floor (default: 1e-8).
.TP
.BI \-\-lr\-schedule " " {plateau,exponential,adaptive}
Learning rate decay schedule (default: \fBplateau\fR).
.TP
.BI \-\-scale\-optimization " " {fixed,iterative,joint}
Block scale tuning strategy: \fBfixed\fR, \fBiterative\fR, or \fBjoint\fR (default: \fBjoint\fR).
.TP
.BI \-\-scale\-refinement\-rounds " " N
Scale refresh iterations when using \fBiterative\fR scale optimization (default: 2).
.TP
.B \-\-full\-matrix
Use full SVD instead of low-rank SVD for reconstruction objective.
.TP
.BI \-\-top\-p " " P ", " \-\-min\-k " " K ", " \-\-max\-k " " K
Singular value decomposition rank selection bounds.
.TP
.BI \-\-target\-loss " " TARGET
Target reconstruction loss threshold for early exit (default: 1e-5; 0 disables).
.TP
.BI \-\-early\-stop\-stall " " STALL
Early stop threshold in steps without loss reduction (default: 2000).
.TP
.BI \-\-early\-stop\-floor\-stall " " FLOOR
Steps without improvement after reaching \fB\-\-lr\-min\fR before terminating matrix search (default: 64; 0 disables).
.TP
.BI \-\-calib\-samples " " SAMPLES
Random calibration vectors used for sibling bias correction (default: 3072).
.TP
.BI \-\-seed " " SEED
Random seed (default: 42).
.TP
.B \-\-no\-bias\-correction
Disable sibling bias correction.
.SS System & Execution
.TP
.BI \-\-device " " DEVICE
Compute device: \fBauto\fR, \fBcpu\fR, \fBcuda\fR, or \fBcuda:N\fR (default: \fBauto\fR).
.TP
.BI \-\-llama\-cpp " " PATH
Path to local llama.cpp repository checkout (default: \fI~/llama.cpp\fR).
.TP
.BI \-\-convert\-to\-quant " " PATH
Path to local convert_to_quant repository checkout.
.TP
.BI \-\-checkpoint\-dir " " DIR
Directory for per-tensor resume checkpoints.
.TP
.B \-\-fresh
Delete checkpoint directory before starting.
.TP
.B \-\-keep\-checkpoints
Keep per-tensor checkpoints on disk after completion.
.TP
.B \-\-eager
Map and load tensors up front.
.TP
.B \-\-no\-temp\-file
Keep finished GGUF tensors in RAM rather than a temporary file.
.TP
.B \-\-verbose
Enable verbose logging.
.TP
.B \-\-self\-test
Verify NVFP4 packing against llama.cpp dequantizer and exit.
.SH EXAMPLES
.TP
.B Quantize an F16 GGUF to NVFP4:
.RS
.nf
gguf_to_nvfp4_gguf -o ./llama3-nvfp4 ./llama-3-8b-f16.gguf
.fi
.RE
.TP
.B Fast conversion with simple grid rounding:
.RS
.nf
gguf_to_nvfp4_gguf --simple -o ./qwen-nvfp4 ./qwen2.5-7b-bf16.gguf
.fi
.RE
.TP
.B Repack existing NVFP4 GGUF into a new directory with Modelfile:
.RS
.nf
gguf_to_nvfp4_gguf --no-quant --outtype nvfp4 -o ./repacked ./model-nvfp4.gguf
.fi
.RE
.SH EXIT STATUS
.TP
.B 0
Success.
.TP
.B 1
An error occurred during quantization or repacking.
.SH SEE ALSO
.BR hf_to_nvfp4_gguf (1),
.BR safetensors_to_nvfp4_gguf (1),
.BR gguf_info (1),
.BR ollama_nvfp4 (1)