manfiles and new scripts

man/man1/ctq.1

@@ -0,0 +1,170 @@
+.TH CTQ 1 "November 2026" "convert_to_quant" "User Commands"
+.SH NAME
+ctq \- convert safetensors model weights to scaled FP8, NVFP4, INT8, and MXFP8 formats
+.SH SYNOPSIS
+.B ctq
+\fB\-i\fR \fIINPUT\fR
+[\fB\-o\fR \fIOUTPUT\fR]
+[\fB\-\-comfy_quant\fR]
+[\fB\-\-simple\fR]
+[\fB\-\-device\fR \fIDEVICE\fR]
+[\fB\-\-low\-memory\fR]
+[\fB\-\-nvfp4\fR|\fB\-\-int8\fR|\fB\-\-mxfp8\fR]
+[\fIoptions...\fR]
+.SH DESCRIPTION
+.B ctq
+(\fBconvert_to_quant\fR) quantizes neural network weights stored in \fI.safetensors\fR
+format into scaled lower-precision representations, primarily targeting
+\fBComfyUI\fR and hardware-accelerated inference runtimes.
+.PP
+By default, \fBctq\fR performs Scaled FP8 quantization using SVD-guided learned rounding
+(Prodigy optimizer) and per-tensor scaling. It supports multiple quantization formats:
+.IP \(bu 2
+\fBFP8\fR (Float8 E4M3 / E5M2) with tensor-wise, row-wise, or block-wise scaling.
+.IP \(bu 2
+\fBNVFP4\fR (NVIDIA 4-bit floating point, E2M1 with dual-level block scaling).
+.IP \(bu 2
+\fBINT8\fR block-wise quantization with optional group-wise Hadamard rotation (\fIConvRot\fR).
+.IP \(bu 2
+\fBMXFP8\fR (Microscaling FP8) and Hybrid MXFP8 formats.
+.PP
+Quantization error can optionally be extracted into a companion Error Correction LoRA
+adapter (\fB\-\-extract\-lora\fR) to restore high-precision fidelity.
+.SH OPTIONS
+.SS Standard Options
+.TP
+.BI \-i " " INPUT ", " \-\-input " " INPUT
+Path to the input \fI.safetensors\fR weight file (required).
+.TP
+.BI \-o " " OUTPUT ", " \-\-output " " OUTPUT
+Path to the output \fI.safetensors\fR destination file. Auto-generated if omitted.
+.TP
+.BR \-\-comfy_quant ", " \-\-comfy\-quant
+Write ComfyUI-compatible quantization tensors and metadata (\fI.comfy_quant\fR).
+.TP
+.B \-\-simple
+Skip SVD learned rounding and apply fast, direct grid quantization.
+.TP
+.BR \-fpmm ", " \-\-full_precision_matrix_mult
+Add \fIfull_precision_matrix_mult=True\fR to \fI.comfy_quant\fR metadata.
+.TP
+.B \-\-heur
+Skip layers with poor quantization characteristics (aspect ratio, size).
+.TP
+.BI \-\-device " " DEVICE
+Execution device (e.g. \fBcuda\fR, \fBcuda:0\fR, \fBcpu\fR). Overrides auto-detection.
+.TP
+.BR \-lm ", " \-\-low\-memory
+Stream tensor loading to minimize RAM consumption (recommended for models exceeding 50% available memory).
+.TP
+.BI \-\-verbose " " {DEBUG,VERBOSE,NORMAL,MINIMAL}
+Logging verbosity level (default: \fBNORMAL\fR).
+.TP
+.BI \-\-manual_seed " " SEED
+Random seed for reproducibility. Set to \-1 for non-deterministic execution.
+.TP
+.B \-\-save\-quant\-metadata
+Write quantization metadata directly into safetensors file header (\fI_quantization_metadata\fR).
+.SS Quantization Formats & Modes
+.TP
+.B \-\-nvfp4
+Quantize to NVIDIA FP4 (E2M1) with dual-level block scaling (Blackwell architecture).
+.TP
+.B \-\-int8
+Quantize to INT8 block-wise format instead of FP8.
+.TP
+.B \-\-convrot
+Enable group-wise Hadamard rotation (ConvRot) for INT8 row-wise quantization to mitigate activation outliers.
+.TP
+.BI \-\-convrot\-group\-size " " SIZE
+Group size for ConvRot (must be a power of 4, e.g. 64, 256, 1024; default: 256).
+.TP
+.B \-\-mxfp8
+Quantize to MXFP8 (Microscaling FP8) format.
+.TP
+.BI \-\-scaling_mode " " {tensor,row,block}
+FP8 scaling granularity: \fBtensor\fR (single scale), \fBrow\fR (per-row), or \fBblock\fR (2D tile). Default: \fBtensor\fR.
+.TP
+.BI \-\-block_size " " SIZE
+Block size for block-wise scaling (default: 128).
+.SS Learned Rounding & Optimization
+.TP
+.BI \-\-optimizer " " {prodigy,adamw,radam,original}
+Optimization algorithm for learned rounding (default: \fBprodigy\fR).
+.TP
+.BI \-\-num_iter " " N
+Optimization iterations per tensor (default: 4000).
+.TP
+.BI \-\-lr " " LR
+Initial learning rate for AdamW/RAdam/Original (default: 1.0).
+.TP
+.BI \-\-lr_schedule " " {plateau,exponential,adaptive}
+Learning rate decay schedule (default: \fBplateau\fR).
+.TP
+.B \-\-use_speed
+Enable Prodigy's \fIuse_speed\fR acceleration parameter.
+.TP
+.BI \-\-top_p " " P
+Cumulative energy proportion of principal components for SVD (default: 0.2).
+.TP
+.BI \-\-min_k " " K ", " \-\-max_k " " K
+Minimum and maximum SVD rank bounds (default: 256 and 1280).
+.TP
+.B \-\-full_matrix
+Compute full SVD instead of low-rank SVD.
+.TP
+.BI \-\-scale\-optimization " " {fixed,iterative,joint,dualround}
+Block scale tuning strategy (default: \fBfixed\fR).
+.SS Error Correction LoRA
+.TP
+.B \-\-extract\-lora
+Extract quantization residual error into a separate LoRA adapter file.
+.TP
+.BI \-\-lora\-rank " " RANK
+Rank for extracted LoRA layers (default: 32).
+.TP
+.BI \-\-lora\-target " " REGEX
+Regex matching layers to target for LoRA extraction.
+.TP
+.BI \-\-lora\-output " " PATH
+Output file path for the extracted LoRA adapter (\fI.safetensors\fR).
+.SS Model Filters
+.B ctq
+includes pre-tuned layer exclusion filters for common architecture families:
+.TP
+.BR \-\-gemma4 ", " \-\-qwen_vlm ", " \-\-t5xxl ", " \-\-wan ", " \-\-hunyuan ", " \-\-ltxv2 ", " \-\-qwen ", " \-\-zimage
+Keep architecture-critical layers (embeddings, layernorms, vision towers, cross-attention) in native precision.
+.SH EXAMPLES
+.TP
+.B Standard FP8 quantization for ComfyUI:
+.RS
+.nf
+ctq -i model.safetensors -o model-fp8.safetensors --comfy_quant
+.fi
+.RE
+.TP
+.B NVFP4 quantization using learned rounding:
+.RS
+.nf
+ctq -i model.safetensors -o model-nvfp4.safetensors --nvfp4 --scale-optimization joint
+.fi
+.RE
+.TP
+.B Fast INT8 conversion with ConvRot outlier mitigation:
+.RS
+.nf
+ctq -i model.safetensors --int8 --convrot --simple -o model-int8.safetensors
+.fi
+.RE
+.TP
+.B Quantize with Error Correction LoRA extraction:
+.RS
+.nf
+ctq -i model.safetensors --nvfp4 --extract-lora --lora-rank 32 --lora-output model-ec-lora.safetensors
+.fi
+.RE
+.SH SEE ALSO
+.BR hf_to_nvfp4_safetensors (1),
+.BR safetensors_to_nvfp4_gguf (1),
+.BR heretic (1)
+