32 lines
1.3 KiB
Markdown
32 lines
1.3 KiB
Markdown
These scripts require [llama.cpp](https://github.com/ggml-org/llama.cpp) and [Convert-to-Quant](https://github.com/silveroxides/convert_to_quant).
|
|
|
|
`hf`, the Hugging Face CLI command is also suggested, but not required.
|
|
|
|
For quantizations or conversions, try:
|
|
- `hf_to_nvfp4_quant` (huggingface structure to GGUF)
|
|
- `gguf_to_nvfp4_quant` (GGUF to GGUF)
|
|
- `safetensors_to_nvfp4_quant` (ComfyUI style safetensors to GGUF (untested))
|
|
- `hf_to_nvfp4_safetensors` (huggingface structure to ComfyUI style safetensors)
|
|
|
|
For model info, try:
|
|
- `hf_info` (huggingface structure)
|
|
- `gguf_info` (GGUF structure)
|
|
---
|
|
|
|
Example:
|
|
|
|
```bash
|
|
hf download coder3101/gemma-4-12B-it-heretic --local-dir gemma-4-12B-it-heretic
|
|
hf_to_nvfp4_gguf -o gemma-4-12B-it-heretic_nvfp4 gemma-4-12B-it-heretic
|
|
```
|
|
|
|
This would create a NVFP4 GGUF for Ollama/llama.cpp from `https://huggingface.co/coder3101/gemma-4-12B-it-heretic`
|
|
|
|
The same directory can be quantized to one ComfyUI NVFP4 safetensors file:
|
|
|
|
`hf_to_nvfp4_safetensors -o gemma-4-12B-it-heretic-NVFP4.safetensors gemma-4-12B-it-heretic`
|
|
|
|
Conversions between formats without quantization is available in the custom commands with `--no-quant`.
|
|
|
|
Currently the only supported modes are "Quantize to NVFP4" or "Copy without quantizing" (`--no-quant`). I may add other quantization options in the future.
|