154 lines
4.7 KiB
Groff
154 lines
4.7 KiB
Groff
.TH CONVERT_LORA_TO_GGUF 1 "November 2026" "nvfp4_gguf_quant" "User Commands"
|
|
.SH NAME
|
|
convert_lora_to_gguf \- convert Hugging Face PEFT LoRA adapters to GGUF format
|
|
.SH SYNOPSIS
|
|
.B convert_lora_to_gguf
|
|
[\fB\-\-outfile\fR \fIOUTFILE\fR]
|
|
[\fB\-\-outtype\fR {\fBf32\fR|\fBf16\fR|\fBbf16\fR|\fBq8_0\fR|\fBauto\fR}]
|
|
[\fB\-\-base\fR \fIBASE\fR]
|
|
[\fB\-\-base\-model\-id\fR \fIID\fR]
|
|
[\fB\-\-trust\-remote\-code\fR]
|
|
[\fB\-\-bigendian\fR]
|
|
[\fB\-\-no\-lazy\fR]
|
|
[\fB\-\-dry\-run\fR]
|
|
[\fB\-\-verbose\fR]
|
|
[\fB\-h\fR|\fB\-\-help\fR]
|
|
\fIlora_path\fR
|
|
.SH DESCRIPTION
|
|
.B convert_lora_to_gguf
|
|
converts fine-tuned Hugging Face PEFT LoRA (Low-Rank Adaptation) adapter checkpoints
|
|
into GGUF adapter files compatible with \fBllama.cpp\fR runtime adapter injection
|
|
(\fB\-\-lora\fR) and offline merging (\fBllama\-export\-lora\fR).
|
|
.PP
|
|
LoRA checkpoints store low-rank weight matrices \fIA\fR and \fIB\fR rather than full model
|
|
weights. To convert these tensors correctly into GGUF format with proper tensor names,
|
|
dimensions, and architecture attributes,
|
|
.B convert_lora_to_gguf
|
|
requires access to the configuration of the base model on which the adapter was trained.
|
|
Base model metadata is automatically retrieved from Hugging Face Hub using the
|
|
\fIbase_model_name_or_path\fR key inside \fIadapter_config.json\fR, or supplied locally via
|
|
\fB\-\-base\fR.
|
|
.PP
|
|
The tool also features full support for:
|
|
.IP \(bu 2
|
|
\fBMergekit LoRAs\fR: Adapters created with \fImergekit\-extract\-lora\fR, including adapted
|
|
token embedding weights and normalization layers.
|
|
.IP \(bu 2
|
|
\fBaLoRA (Activation-aware LoRA)\fR: Automatic tokenization of \fIinvocation_string\fR into
|
|
GGUF \fIadapter.alora_invocation_tokens\fR metadata.
|
|
.IP \(bu 2
|
|
Both SafeTensors (\fIadapter_model.safetensors\fR) and legacy PyTorch
|
|
(\fIadapter_model.bin\fR) checkpoint formats.
|
|
.SH POSITIONAL ARGUMENTS
|
|
.TP
|
|
.I lora_path
|
|
Directory containing the Hugging Face PEFT LoRA adapter. Must contain
|
|
\fIadapter_config.json\fR along with \fIadapter_model.safetensors\fR or
|
|
\fIadapter_model.bin\fR.
|
|
.SH OPTIONS
|
|
.TP
|
|
.BI \-\-outfile " " OUTFILE
|
|
Destination path for the output GGUF file. Defaults to an automatically named file
|
|
inside the adapter directory. If the substring \fI{ftype}\fR is present in the path, it is
|
|
dynamically replaced with the chosen output format name.
|
|
.TP
|
|
.BI \-\-outtype " " TYPE
|
|
Output tensor precision format. Choices are:
|
|
.RS
|
|
.TP
|
|
.B f32
|
|
(Default) 32-bit single-precision floating point. Recommended for preserving adapter
|
|
fidelity.
|
|
.TP
|
|
.B f16
|
|
16-bit half-precision IEEE float.
|
|
.TP
|
|
.B bf16
|
|
16-bit Brain Floating Point.
|
|
.TP
|
|
.B q8_0
|
|
Quantize adapter weights to 8-bit integer format.
|
|
.TP
|
|
.B auto
|
|
Select the highest-fidelity 16-bit floating point format based on the first loaded
|
|
tensor.
|
|
.RE
|
|
.TP
|
|
.BI \-\-base " " BASE
|
|
Path to a local directory containing configuration files (\fIconfig.json\fR,
|
|
\fItokenizer.json\fR) of the base model. Note that only configuration and tokenizer
|
|
files are required; full model weights are not loaded.
|
|
.TP
|
|
.BI \-\-base\-model\-id " " ID
|
|
Hugging Face model repository ID of the base model (e.g.
|
|
\fImeta-llama/Llama-3.2-1B-Instruct\fR). If provided, overrides \fB\-\-base\fR and
|
|
fetches configuration directly from Hugging Face Hub.
|
|
.TP
|
|
.B \-\-trust\-remote\-code
|
|
Permit execution of custom modeling and tokenizer code hosted in remote Hugging Face
|
|
repositories when downloading base model configuration.
|
|
.TP
|
|
.B \-\-bigendian
|
|
Target big-endian architecture when serializing tensor headers and raw data.
|
|
.TP
|
|
.B \-\-no\-lazy
|
|
Disable lazy tensor loading, computing and staging all modified tensors in memory
|
|
before writing.
|
|
.TP
|
|
.B \-\-dry\-run
|
|
Simulate the conversion process and log all planned tensor operations and metadata
|
|
without writing any output files to disk.
|
|
.TP
|
|
.B \-\-verbose
|
|
Enable verbose debug logging output.
|
|
.TP
|
|
.BR \-h ", " \-\-help
|
|
Display command usage summary and option descriptions, then exit.
|
|
.SH EXAMPLES
|
|
.TP
|
|
.B Convert a local LoRA adapter with automatic base model download:
|
|
.RS
|
|
.nf
|
|
convert_lora_to_gguf ./my-llama3-lora/
|
|
.fi
|
|
.RE
|
|
.TP
|
|
.B Convert a LoRA adapter using a local base model directory:
|
|
.RS
|
|
.nf
|
|
convert_lora_to_gguf --base ./Meta-Llama-3-8B/ ./my-llama3-lora/
|
|
.fi
|
|
.RE
|
|
.TP
|
|
.B Convert with explicit output path in 16-bit float:
|
|
.RS
|
|
.nf
|
|
convert_lora_to_gguf --outtype f16 --outfile ./llama3-adapter-f16.gguf \\
|
|
./my-llama3-lora/
|
|
.fi
|
|
.RE
|
|
.TP
|
|
.B Use the converted adapter in llama.cpp:
|
|
.RS
|
|
.nf
|
|
llama-cli -m llama-3-8b-Q4_K_M.gguf --lora llama3-adapter-f16.gguf \\
|
|
-p "Hello, world!"
|
|
.fi
|
|
.RE
|
|
.SH EXIT STATUS
|
|
.TP
|
|
.B 0
|
|
Success.
|
|
.TP
|
|
.B 1
|
|
Conversion failed (e.g. missing adapter files, base model configuration mismatch, or
|
|
unsupported architecture).
|
|
.SH SEE ALSO
|
|
.BR convert_hf_to_gguf (1),
|
|
.BR convert_hf_to_gguf_update (1),
|
|
.BR convert_llama_ggml_to_gguf (1),
|
|
.BR gguf_to_nvfp4_gguf (1),
|
|
.BR hf_to_nvfp4_gguf (1),
|
|
.BR gguf_info (1)
|
|
|