add scripts/manpages for llama.cpp converters
This commit is contained in:
zefie committed
1 parent
66d47eea6a
commit
0ef586d131
12 files changed
+667
No files matched your search
@@ -0,0 +1,153 @@
|
||||
.TH CONVERT_LORA_TO_GGUF 1 "November 2026" "nvfp4_gguf_quant" "User Commands"
|
||||
.SH NAME
|
||||
convert_lora_to_gguf \- convert Hugging Face PEFT LoRA adapters to GGUF format
|
||||
.SH SYNOPSIS
|
||||
.B convert_lora_to_gguf
|
||||
[\fB\-\-outfile\fR \fIOUTFILE\fR]
|
||||
[\fB\-\-outtype\fR {\fBf32\fR|\fBf16\fR|\fBbf16\fR|\fBq8_0\fR|\fBauto\fR}]
|
||||
[\fB\-\-base\fR \fIBASE\fR]
|
||||
[\fB\-\-base\-model\-id\fR \fIID\fR]
|
||||
[\fB\-\-trust\-remote\-code\fR]
|
||||
[\fB\-\-bigendian\fR]
|
||||
[\fB\-\-no\-lazy\fR]
|
||||
[\fB\-\-dry\-run\fR]
|
||||
[\fB\-\-verbose\fR]
|
||||
[\fB\-h\fR|\fB\-\-help\fR]
|
||||
\fIlora_path\fR
|
||||
.SH DESCRIPTION
|
||||
.B convert_lora_to_gguf
|
||||
converts fine-tuned Hugging Face PEFT LoRA (Low-Rank Adaptation) adapter checkpoints
|
||||
into GGUF adapter files compatible with \fBllama.cpp\fR runtime adapter injection
|
||||
(\fB\-\-lora\fR) and offline merging (\fBllama\-export\-lora\fR).
|
||||
.PP
|
||||
LoRA checkpoints store low-rank weight matrices \fIA\fR and \fIB\fR rather than full model
|
||||
weights. To convert these tensors correctly into GGUF format with proper tensor names,
|
||||
dimensions, and architecture attributes,
|
||||
.B convert_lora_to_gguf
|
||||
requires access to the configuration of the base model on which the adapter was trained.
|
||||
Base model metadata is automatically retrieved from Hugging Face Hub using the
|
||||
\fIbase_model_name_or_path\fR key inside \fIadapter_config.json\fR, or supplied locally via
|
||||
\fB\-\-base\fR.
|
||||
.PP
|
||||
The tool also features full support for:
|
||||
.IP \(bu 2
|
||||
\fBMergekit LoRAs\fR: Adapters created with \fImergekit\-extract\-lora\fR, including adapted
|
||||
token embedding weights and normalization layers.
|
||||
.IP \(bu 2
|
||||
\fBaLoRA (Activation-aware LoRA)\fR: Automatic tokenization of \fIinvocation_string\fR into
|
||||
GGUF \fIadapter.alora_invocation_tokens\fR metadata.
|
||||
.IP \(bu 2
|
||||
Both SafeTensors (\fIadapter_model.safetensors\fR) and legacy PyTorch
|
||||
(\fIadapter_model.bin\fR) checkpoint formats.
|
||||
.SH POSITIONAL ARGUMENTS
|
||||
.TP
|
||||
.I lora_path
|
||||
Directory containing the Hugging Face PEFT LoRA adapter. Must contain
|
||||
\fIadapter_config.json\fR along with \fIadapter_model.safetensors\fR or
|
||||
\fIadapter_model.bin\fR.
|
||||
.SH OPTIONS
|
||||
.TP
|
||||
.BI \-\-outfile " " OUTFILE
|
||||
Destination path for the output GGUF file. Defaults to an automatically named file
|
||||
inside the adapter directory. If the substring \fI{ftype}\fR is present in the path, it is
|
||||
dynamically replaced with the chosen output format name.
|
||||
.TP
|
||||
.BI \-\-outtype " " TYPE
|
||||
Output tensor precision format. Choices are:
|
||||
.RS
|
||||
.TP
|
||||
.B f32
|
||||
(Default) 32-bit single-precision floating point. Recommended for preserving adapter
|
||||
fidelity.
|
||||
.TP
|
||||
.B f16
|
||||
16-bit half-precision IEEE float.
|
||||
.TP
|
||||
.B bf16
|
||||
16-bit Brain Floating Point.
|
||||
.TP
|
||||
.B q8_0
|
||||
Quantize adapter weights to 8-bit integer format.
|
||||
.TP
|
||||
.B auto
|
||||
Select the highest-fidelity 16-bit floating point format based on the first loaded
|
||||
tensor.
|
||||
.RE
|
||||
.TP
|
||||
.BI \-\-base " " BASE
|
||||
Path to a local directory containing configuration files (\fIconfig.json\fR,
|
||||
\fItokenizer.json\fR) of the base model. Note that only configuration and tokenizer
|
||||
files are required; full model weights are not loaded.
|
||||
.TP
|
||||
.BI \-\-base\-model\-id " " ID
|
||||
Hugging Face model repository ID of the base model (e.g.
|
||||
\fImeta-llama/Llama-3.2-1B-Instruct\fR). If provided, overrides \fB\-\-base\fR and
|
||||
fetches configuration directly from Hugging Face Hub.
|
||||
.TP
|
||||
.B \-\-trust\-remote\-code
|
||||
Permit execution of custom modeling and tokenizer code hosted in remote Hugging Face
|
||||
repositories when downloading base model configuration.
|
||||
.TP
|
||||
.B \-\-bigendian
|
||||
Target big-endian architecture when serializing tensor headers and raw data.
|
||||
.TP
|
||||
.B \-\-no\-lazy
|
||||
Disable lazy tensor loading, computing and staging all modified tensors in memory
|
||||
before writing.
|
||||
.TP
|
||||
.B \-\-dry\-run
|
||||
Simulate the conversion process and log all planned tensor operations and metadata
|
||||
without writing any output files to disk.
|
||||
.TP
|
||||
.B \-\-verbose
|
||||
Enable verbose debug logging output.
|
||||
.TP
|
||||
.BR \-h ", " \-\-help
|
||||
Display command usage summary and option descriptions, then exit.
|
||||
.SH EXAMPLES
|
||||
.TP
|
||||
.B Convert a local LoRA adapter with automatic base model download:
|
||||
.RS
|
||||
.nf
|
||||
convert_lora_to_gguf ./my-llama3-lora/
|
||||
.fi
|
||||
.RE
|
||||
.TP
|
||||
.B Convert a LoRA adapter using a local base model directory:
|
||||
.RS
|
||||
.nf
|
||||
convert_lora_to_gguf --base ./Meta-Llama-3-8B/ ./my-llama3-lora/
|
||||
.fi
|
||||
.RE
|
||||
.TP
|
||||
.B Convert with explicit output path in 16-bit float:
|
||||
.RS
|
||||
.nf
|
||||
convert_lora_to_gguf --outtype f16 --outfile ./llama3-adapter-f16.gguf \\
|
||||
./my-llama3-lora/
|
||||
.fi
|
||||
.RE
|
||||
.TP
|
||||
.B Use the converted adapter in llama.cpp:
|
||||
.RS
|
||||
.nf
|
||||
llama-cli -m llama-3-8b-Q4_K_M.gguf --lora llama3-adapter-f16.gguf \\
|
||||
-p "Hello, world!"
|
||||
.fi
|
||||
.RE
|
||||
.SH EXIT STATUS
|
||||
.TP
|
||||
.B 0
|
||||
Success.
|
||||
.TP
|
||||
.B 1
|
||||
Conversion failed (e.g. missing adapter files, base model configuration mismatch, or
|
||||
unsupported architecture).
|
||||
.SH SEE ALSO
|
||||
.BR convert_hf_to_gguf (1),
|
||||
.BR convert_hf_to_gguf_update (1),
|
||||
.BR convert_llama_ggml_to_gguf (1),
|
||||
.BR gguf_to_nvfp4_gguf (1),
|
||||
.BR hf_to_nvfp4_gguf (1),
|
||||
.BR gguf_info (1)
|
||||
|
||||
Reference in new issue
Block a user