include llama.cpp since there were some edits
This commit is contained in:
zefie committed
1 parent
db7f28d087
commit
8c66990853
3918 files changed
+1640170
-1
No files matched your search
@@ -0,0 +1,99 @@
|
||||
---
|
||||
name: add-new-model
|
||||
description: Guided workflow for adding a new model architecture to llama.cpp. Use when the user wants to add/port a new model architecture.
|
||||
---
|
||||
|
||||
# Add a new model architecture to llama.cpp
|
||||
|
||||
This skill walks a contributor through adding a new model architecture. AI-generated code is permitted in this project, so you may write full implementations for the steps below rather than only pointing at patterns - but follow `AGENTS.md`'s AI usage policy throughout:
|
||||
|
||||
- The contributor is 100% responsible for every line, however it was produced. They must be able to explain and defend any part of it to a reviewer. Check in with them as you go (don't silently generate everything and hand over a finished diff) so they actually absorb what was written.
|
||||
- Before writing code, make sure the contributor owns the design choices for this architecture (which reference model to follow, how non-standard bits like RoPE variants or MoE routing should be handled) - AI accelerates a design the contributor has already made, it doesn't make the design for them.
|
||||
- Disclosure is mandatory: any AI-meaningful contribution must be disclosed per the PR template. Remind the contributor of this before they open the PR.
|
||||
- Never write the PR description, commit message, GitHub issue/discussion post, or reviewer replies - those must come from the contributor. If asked to commit on their behalf, use `Assisted-by:` (never `Co-authored-by:`) and only after explicit confirmation.
|
||||
- If the requested change looks large or introduces a new pattern not covered here, pause and tell the user this kind of change is likely to need prior discussion with maintainers before a PR.
|
||||
- Keep the PR self-contained. If the work would require a lot of unconventional changes outside the new model file(s) (e.g. touching shared graph-building code, the sampler, or core APIs in ways other models don't), STOP and tell the contributor to open a discussion/issue first - invasive or excessive changes get closed without full review.
|
||||
- Do not bundle unrelated work into this PR - see Step 4 and Step 5 below for the specifics on multimodal and chat-template/parsing work.
|
||||
- Never hack around RoPE with a custom sin/cos implementation. Several past PRs tried this and were closed. If the existing `ggml_rope_ext` (see Step 2's RoPE tips) genuinely cannot express what this model needs, the contributor should open an issue to discuss it with maintainers first - not send a PR with a custom RoPE implementation.
|
||||
|
||||
Before starting, read `CONTRIBUTING.md`, `AGENTS.md` and `docs/development/HOWTO-add-model.md` if they are not already in context. Also run `git log --oneline -- src/models` and look at at least 3 recent PRs that added a model (their merge commits/diffs) - this shows current convention more reliably than the docs, which can lag behind.
|
||||
|
||||
## Step 0 - Scope and dedup check
|
||||
|
||||
Ask the contributor:
|
||||
1. Which model (HF repo id or name)? Is it text-only or does it have a multimodal (vision/audio) encoder?
|
||||
2. Do they already have the HF `config.json`/weights available locally?
|
||||
3. Have they checked for an existing PR/issue on this model? Suggest `gh search issues "<model name>"` and `gh search prs "<model name>"` in the `ggml-org/llama.cpp` repo. If an existing PR covers it, the contributor should comment there and collaborate rather than open a duplicate (per CONTRIBUTING.md's AI Usage Policy).
|
||||
4. What existing supported architecture is this model closest to (e.g. "Llama-like with sliding window", "MoE like DBRX", "BERT-style encoder")?
|
||||
|
||||
If the contributor doesn't know the closest reference architecture, you may grep `conversion/*.py` and `src/models/*.cpp` for architectures with a similar config shape (layer count, head count, MoE expert count, norm placement) and suggest 1-2 candidates - but let the contributor confirm the choice rather than picking one yourself; this choice is a design decision they need to own.
|
||||
|
||||
Do not proceed to Step 1 until the contributor has answered these and named a reference architecture.
|
||||
|
||||
## Step 1 - Convert the model to GGUF
|
||||
|
||||
Follow HOWTO-add-model.md section 1 for the actual touch points (conversion class registration, `constants.py`, `tensor_mapping.py`, etc.) - don't re-derive them here, read them from the doc.
|
||||
|
||||
Skill-specific addition: for each touch point, show the contributor the equivalent code in the reference architecture they named in Step 0 before writing the new version, and check that they understand what's different about their model (e.g. non-standard tensor shapes, extra hparams) rather than just copying the pattern silently.
|
||||
|
||||
## Step 2 - Define the architecture in llama.cpp
|
||||
|
||||
Follow HOWTO-add-model.md section 2 for the actual touch points (`llm_arch` enum, `LLM_ARCH_NAMES`, hparam loading, RoPE type case, etc.), including its "Tips and tricks" section for `ggml_rope_ext` gotchas.
|
||||
|
||||
Skill-specific addition: never hack around RoPE with a custom sin/cos implementation - see the RoPE rule above.
|
||||
|
||||
## Step 3 - Build the GGML graph
|
||||
|
||||
Follow HOWTO-add-model.md section 3 for the actual touch points (`src/models/<name>.cpp` struct, `llama_model_mapping` registration, etc.).
|
||||
|
||||
Skill-specific addition: before writing `src/models/<name>.cpp`, read at least 10 other files under `src/models/` (pick a mix, not just the one reference architecture) to confirm the struct layout, naming, and style you're about to write actually matches current convention - the pattern drifts over time and the HOWTO doc can lag behind it.
|
||||
|
||||
## Step 4 - Optional: multimodal encoder
|
||||
|
||||
Only do this if the contributor flagged a vision/audio encoder in Step 0. Follow HOWTO-add-model.md section 4 and `docs/multimodal.md` for the actual touch points (`MmprojModel` subclass, `clip.cpp`, `mtmd.cpp`, encoder graph in `tools/mtmd/models`, etc.).
|
||||
|
||||
Skill-specific addition, and read this carefully: **whether the multimodal encoder can be bundled into the same PR as the base text-model support depends on how conventional the change is.** It's OK to bundle it if the encoder support is conventional - i.e. no new infra or logic is needed, it's just a new cgraph reusing existing preprocessing/projector machinery (e.g. siglip/pixtral/qwen with just a new projector). If it requires anything beyond that - a new preprocessor, non-standard projector logic, or changes to shared `libmtmd` infra/logic - STOP, tell the contributor this is non-conventional, and have them land the text model first with the encoder as a dedicated follow-up PR. Do not let this decision pass silently - call it out explicitly to the contributor before writing any `clip.cpp`/`mtmd.cpp` code.
|
||||
|
||||
## Step 5 - Optional: chat template / parsing support
|
||||
|
||||
Only do this if the model needs a new built-in chat template (`src/llama-chat.cpp`) or a new output parser (see `docs/development/parsing.md` and `docs/autoparser.md`). If either is needed beyond what a user-supplied Jinja template already covers, treat it as its own dedicated follow-up PR, not part of the base model-support PR - call this out explicitly to the contributor rather than silently bundling it in.
|
||||
|
||||
## Common pitfalls (from past PR reviews)
|
||||
|
||||
These recur often enough in review comments on past add-model PRs that they're worth checking proactively, not just waiting for a reviewer to catch them:
|
||||
|
||||
- Don't validate the same hparam/config assumption in both the Python conversion script and the C++ load path - pick one layer to own the check, duplicating it just adds maintenance surface.
|
||||
- Optional hparams that are genuinely absent from some configs (e.g. a shared-expert count) should be read with an explicit optional/fallback accessor, not assumed present.
|
||||
- Hparams that are actually load-bearing (the model produces wrong output or crashes without them, e.g. `sliding_window_pattern`, norm-eps) must hard-error if missing, not silently fall back to a default.
|
||||
- Don't bake a default chat template into the C++ binary - inject it into the GGUF at conversion time instead, since one `llm_arch` can be reused by multiple fine-tunes with different templates, and a baked-in C++ default fails silently for those.
|
||||
- Before writing a dedicated tool-call/output parser, check whether the existing autoparser already handles the template (`test-chat-auto-parser <jinja>` shows what it detects).
|
||||
- Marking a custom EOS/closing-tag token as `eot` at conversion time isn't always sufficient - in long/agentic generations a model can emit the closing sequence as literal text instead of the token, so generation never stops on EOG and raw text leaks past the parser. Verify this case, not just the token path.
|
||||
- If reusing or aliasing an existing pre-tokenizer for convenience, justify and test that choice explicitly - silent reuse is an easy source of subtle tokenizer bugs.
|
||||
- Watch for excessive graph splits caused by building per-layer view/index tensors inside the layer loop - hoist tensors that don't vary per layer out of the loop (relevant if you hit `GGML_SCHED_MAX_SPLIT_INPUTS`).
|
||||
- A custom KQ mask fed into flash attention must match FA's expected dtype - cast it to F16 before passing it to `build_attn_mha` when FA is enabled.
|
||||
- When padding a custom KV-cache size to an alignment (e.g. `GGML_PAD(..., 256)`), apply the padding after all other size adjustments, not before - otherwise later logic can un-align it again.
|
||||
- For non-standard cache/SWA (sliding-window-attention) semantics, override the dedicated hook (e.g. `llama_model_n_swa()`) rather than mutating hparams to fake the behavior - hparams may be read elsewhere for unrelated purposes.
|
||||
- Don't ship unfinished or unverified speculative-decoding (e.g. MTP) scaffolding in the base model PR - if it hasn't actually been confirmed to work, pull it out and land it as its own follow-up.
|
||||
- Conversion code should call into the base class's existing hparam logic (e.g. `super().set_gguf_parameters()`) rather than re-deriving it - large blocks of code that duplicate what `TextModel`/`MmprojModel` already provide will get flagged as redundant.
|
||||
- Do constant tensor modifications (e.g. `norm(1 + weight)`) and permutations/chunking at conversion time, not in the graph - see HOWTO-add-model.md's "Prefer conversion-time tensor modifications" tip (Gemma 3 folds its `1 +` into the weights, Qwen3-Next permutes in `modify_tensors`). Doing these at runtime in the graph is very likely to be rejected as over-complicated; if you genuinely can't do it at conversion time, open a discussion first explaining why rather than implementing it in the graph.
|
||||
- Exception: a plain `weight * scale` with a constant scale is usually better applied at inference time instead of being folded into the weight at conversion. The scale conceptually applies to the activation, not the weight, so folding it in can hurt numerical stability, and it shifts the weight's value range in a way that can make quantization worse.
|
||||
|
||||
## Validation checklist
|
||||
|
||||
Reference: `examples/model-conversion/README.md`.
|
||||
|
||||
1. Convert to GGUF, then inspect/run both the original and converted tensors.
|
||||
2. Run logits verification (original vs converted). If this model is a new version of an already-supported family, verify the *previous* version still passes logits verification first - numerical differences may be pre-existing, not caused by the new work. The tools to perform full logits validation are available in `examples/model-conversion`.
|
||||
3. Quantize (including QAT variants if relevant) and re-verify.
|
||||
4. Run perplexity evaluation (simple and full).
|
||||
5. Sanity-check across `tools/cli`, `tools/completion`, `tools/imatrix`, `tools/quantize`, and `tools/server`.
|
||||
6. CPU backend first; other backends (CUDA, Metal, ...) can be separate follow-up PRs per `CONTRIBUTING.md`.
|
||||
7. Re-review every changed file against the coding/naming guidelines in `AGENTS.md` (and `CONTRIBUTING.md`'s "Coding guidelines"/"Naming guidelines" sections) - this is a separate pass from functional testing and is just as important: no forced line-wrapping, no unicode punctuation, minimal/non-redundant comments, `snake_case` naming (`kebab-case` for file names), matching indentation/brace style, etc.
|
||||
|
||||
## Before opening a PR
|
||||
|
||||
- Run the `code-review` skill on the diff first - it catches the convention and scope issues reviewers flag most often, and it's recommended to do this locally before pushing the PR.
|
||||
- Confirm the contributor can explain every changed line to a reviewer and is prepared to be asked about any of it - this is required regardless of how much of the code was AI-generated.
|
||||
- Confirm they did a comprehensive manual review of the full diff, not just a skim.
|
||||
- Fill in the AI-disclosure section of `.github/pull_request_template.md` describing how AI was used (do not omit or understate this).
|
||||
- Do not write the PR description, commit message, GitHub issue/discussion text, or any reviewer replies yourself - the contributor writes these.
|
||||
@@ -0,0 +1,244 @@
|
||||
---
|
||||
name: code-review
|
||||
description: Review llama.cpp changes against project conventions and common reviewer pitfalls before a PR. Use when the user wants to review a diff, branch, or PR.
|
||||
---
|
||||
|
||||
# Review llama.cpp changes
|
||||
|
||||
This skill reviews changes against llama.cpp's conventions and the pitfalls that reviewers flag most often, so the contributor can fix them before a maintainer has to. It has two modes:
|
||||
|
||||
- **Self-review (default):** review the contributor's own local changes (uncommitted work, or a branch vs `master`) as a pre-PR pass. Ask which if it's ambiguous; default to `git diff master...HEAD` plus any uncommitted changes.
|
||||
- **Read-only review of a PR/file:** if the user points at a PR number or specific files (including code they didn't write), review those and report findings.
|
||||
|
||||
In both modes the output is **private review notes for the user to read and act on** - it is never something to post. This is a hard rule from `AGENTS.md`: an agent must NEVER write, or help write, a PR comment, a review comment, or a reply to a reviewer, by any means including `gh`. Do not offer to. If the user asks you to post the notes, refuse and point them at that rule. Present findings in the conversation only.
|
||||
|
||||
Before starting, read `AGENTS.md` and `CONTRIBUTING.md` if not already in context - the "Coding guidelines", "Naming guidelines", and AI usage sections are the baseline this review enforces. For a diff that adds a new model architecture, also read `docs/development/HOWTO-add-model.md` and consider the dedicated `add-new-model` skill.
|
||||
|
||||
## Step 0 - Scope the diff and pick the checklists
|
||||
|
||||
Identify what actually changed and which area checklists below apply. Run `git diff --stat` (or `gh pr view <n> --json files` for PR mode) and bucket the touched paths:
|
||||
|
||||
- `conversion/`, `gguf-py/`, `src/models/`, `src/llama-arch.*` -> **New model / architecture**
|
||||
- `ggml/` (any backend, op, or `ggml.h`) -> **ggml / backend**
|
||||
- `include/llama.h` and other public headers -> **Public API**
|
||||
- `tools/server/` -> **Server**
|
||||
- `tools/ui/` (or the legacy `tools/server/webui/`) -> **WebUI**
|
||||
- `tests/`, `tools/server/tests/`, or any other added/changed test -> **Tests**
|
||||
- anything else, plus all of the above -> **General** (always runs)
|
||||
|
||||
Always run the **Scope and quick-reject gate**, the **Security review**, and the **General** checklist. Run each area checklist whose paths were touched. Additionally, if the diff introduces a new component, subsystem, or piece of infrastructure (a new file/class/module, a new abstraction, or hand-rolled machinery), run the **Approach and design** review. Tell the user which checklists you're running and why.
|
||||
|
||||
## Scope and quick-reject gate (always)
|
||||
|
||||
These are the patterns that get PRs closed without a full review. Check them first - a finding here is more important than any code nit, because it can mean the change shouldn't be a PR in its current form at all.
|
||||
|
||||
- Is there a prior issue/discussion for this? Features are supposed to start as an issue, not a PR (`CONTRIBUTING.md`). If this is a nontrivial feature with no linked issue, flag it and suggest opening one first.
|
||||
- Is it a duplicate of existing/in-flight work? Suggest `gh search prs` / `gh search issues` for the feature. Many closed PRs were duplicates of something already queued.
|
||||
- Is it self-contained and single-purpose? Multiple unrelated changes/optimizations bundled together get sent back to be split. Flag unrelated changes and suggest separate PRs.
|
||||
- Does it touch multiple ggml backends at once? Initial support should be CPU-only, other backends as follow-ups (`CONTRIBUTING.md`). Flag CUDA/Metal/Vulkan/etc. changes bundled into a feature's first PR.
|
||||
- Does it add a new `ggml_type` / quantization type? That carries a disproportionate maintenance burden and needs the full justification package (GGUF sample upload, perplexity vs FP16/BF16 and similar sizes, KL-divergence data, CPU perf numbers). Absent that, it will be rejected regardless of code quality.
|
||||
- Is it invasive - new subsystem, core-API reshaping, changes to shared graph/sampler code that other models don't need? Flag it and suggest a discussion with maintainers before investing further.
|
||||
- Does it add a model-specific CLI argument to any binary (`common/arg.cpp`, tools, examples) or conversion script (`convert_*.py`)? Not allowed - model-specific behavior must come from GGUF metadata or be detected automatically, not from a per-model flag.
|
||||
- Is it niche/vendor-specific in a way that adds a maintenance burden nobody will own long-term? Flag the maintenance-ownership question.
|
||||
- Is the change semantically correct, or a plausible-looking "fix" that misunderstands the code? Sanity-check the actual behavior, not just that it compiles.
|
||||
- AI-disclosure: if AI meaningfully contributed, is the PR template's disclosure section filled in? Remind the user. Never suggest writing the PR description or commit message for them.
|
||||
|
||||
## Security review (mandatory)
|
||||
|
||||
Mandatory on every review; any finding here is **blocking**. Rule of thumb: GGUF metadata, tensor shapes, tokenizer/grammar input, and all server/RPC fields are attacker-controlled - bound them before use.
|
||||
|
||||
- **Sizes/counts from tensor dims:** validate before allocating. Products like `ne[i]*nb[i]`/nbytes can overflow on crafted dims into an undersized alloc then heap overflow. Overflow checks must run BEFORE the arithmetic they guard - padding/alignment macros wrap to 0 near `SIZE_MAX`, so a guard after the pad passes.
|
||||
- **GGUF strings/arrays:** cap declared lengths and element counts before using them to size a loop or buffer; validate element type and length before casting an array to a pointer or reading fixed indices (`[i+1]`, `[0..2]`).
|
||||
- **Element-type confusion:** casting `gguf_get_arr_data()` or `tensor->data` to `float *`/`int32_t *` needs an element-type check first (`gguf_get_kv_type() == GGUF_TYPE_ARRAY` then `gguf_get_arr_type()`; `type == GGML_TYPE_F32` for tensors). A `UINT8` array or `I8` tensor passes every length check, then gets read 4 bytes per element - a nearby length check is not a type check.
|
||||
- **Loaders:** `GGML_ASSERT` on a file-derived value aborts the process; throw instead where the caller already catches (vocab, model loader, clip).
|
||||
- **File-supplied counts indexing fixed arrays:** bound any count (e.g. layer/block count into a `LLAMA_MAX_*` array) before indexing; watch checks that only fire when an optional key is present.
|
||||
- **Declared vs actual array length:** check the declared length of a GGUF array against the count actually read, not just against a buffer size.
|
||||
- **Bounds comparisons:** flag narrowing casts (`size_t`->`int32_t`) and signed/unsigned mixing that can bypass a length check and copy past a buffer.
|
||||
- **Parsed/derived indices:** range-check `stoi`/`atoi` results and catch parse throws; never use a default or derived token id (EOS/BOS/...) as an index without a bounds check.
|
||||
- **Reused/reserved buffers:** recheck bounds after a buffer is shrunk or reused; watch `reserve()` then index-by-assumed-size, and header fields read before their length is checked.
|
||||
- **Server JSON ints:** clamp client-supplied integers (token/discard counts, offsets) to non-negative and an upper bound before they reach index/pointer arithmetic.
|
||||
- **RPC-deserialized fields:** treat every field (type/buffer/data/ne/nb/op_params) as hostile - validate before use. Null/zero buffers skipping validation, attacker data pointers, out-of-range type indices, and negative strides sign-extending past a corner-only assert all give arbitrary read/write.
|
||||
- **Recursion on untrusted input:** a function that recurses once per level (or per char) of request JSON, model output, a grammar, a template or a regex pattern overflows the stack - the HTTP threads have 512 KB on macOS, so a few hundred levels are enough. Every such function needs its own depth bound, checked in the function itself, not a pre-walk at a caller (the next caller skips it): libcommon uses `COMMON_RECURSIVE_LIMIT` (`common/common.h`), jinja `JINJA_RECURSIVE_LIMIT` (`common/jinja/utils.h`), the grammar `GRAMMAR_RECURSIVE_LIMIT` (`src/llama-grammar.h`). Count the levels of the input, not the frames, and throw `std::invalid_argument` for request input so the server answers 400. Look for recursion the code does not spell out: mutual recursion, recursive `std::function` lambdas, visitor/virtual dispatch, `std::regex` (libstdc++ recurses per char of the input), and per-char helpers that recurse where a loop would do.
|
||||
- **Lifetime/UAF:** flag stored raw pointers to caller/temporary storage, cached pointers to buffers a later free releases, async ops whose source may drop before completion, and structures not invalidated on free/realloc. Null-check conditionally-built or "not required" tensors before dereferencing.
|
||||
|
||||
## Approach and design (when a new component/infra is introduced)
|
||||
|
||||
Run this whenever the diff adds a new component, subsystem, or piece of infrastructure. Reviews too often stop at "does it work" - a diff can be correct and still be the wrong approach, and a messy design costs more long-term than a bug. Evaluate the *approach*, not just the behavior; raising a cleaner one is a high-value finding, not a nit. If you see a better design, describe it concretely rather than just calling the current one bad.
|
||||
|
||||
- **Simpler approach upstream:** the biggest win is often a different data model or design that removes whole subsystems, not tweaks to the code as written. Complexity must be justified by the problem, not by the first thing that worked.
|
||||
- **Reuse over reinvention:** grep for an existing helper, library, object, or mechanism before adding a new one. Reimplementing what the codebase already has reintroduces solved bugs and adds maintenance surface.
|
||||
- **Clear ownership/lifetime:** prefer RAII and obvious ownership over manual liveness flags, hand-tracked pointers, and "is it still alive?" checks - manual lifetime tracking is a recurring source of subtle bugs.
|
||||
- **Right-sized machinery:** flag redundant, overkill, or heavier-than-needed primitives and abstractions; use the minimum the design actually needs.
|
||||
- **Right structure and fit:** a new type should earn its place (split it if it serves two roles); follow existing patterns, idioms, and naming, and avoid constructs the project shuns.
|
||||
- **Root cause vs symptom:** fixes layered on fixes signal a design to correct, not guard around.
|
||||
|
||||
## New model / architecture
|
||||
|
||||
See the `add-new-model` skill and `docs/development/HOWTO-add-model.md` for the full workflow; this is the review-time subset that reviewers most often catch:
|
||||
|
||||
- Don't branch on `model.arch` when the real dependency is a config/capability value - gate on the hparam/capability, not the architecture enum.
|
||||
- If the model is a close variant of an existing arch, is the delta justified? Prefer reusing or subclassing the existing arch/model class over duplicating it. A near-duplicate class or `src/models/<name>.cpp` will be asked to merge with its sibling.
|
||||
- New tensor names go through `tensor_mapping.py`, not ad-hoc name matching.
|
||||
- For QKV, split the *activation* with `ggml_view`, not the *weight* tensor; rely on ggml broadcasting instead of manually duplicating tensors.
|
||||
- New graph inputs are declared at the top of the graph-build function, not inline where first used.
|
||||
- Hparams that the model can't run correctly without must be mandatory (hard-error if missing), not read with a silent default fallback. Only genuinely-optional-across-configs values get a fallback accessor.
|
||||
- New/optional weight tensors (scales, etc.) must route through `build_lora_mm` and the existing helpers, matching convention - don't leave raw matmuls copied from another arch.
|
||||
- Don't hack RoPE with a custom sin/cos implementation. If `ggml_rope_ext` genuinely can't express it, that's an issue for discussion, not a PR.
|
||||
- Test the quantized-KV path (`-ctk`/`-ctv q8_0`), not just default f16 - new speculative/attention features silently break there.
|
||||
- Preserve existing explanatory comments about model-specific quirks when copying code; note the provenance ("copied from X, with Y added").
|
||||
- Remove dead code/branches left over from adapting a reference implementation.
|
||||
|
||||
## ggml / backend
|
||||
|
||||
- `supports_op` (and any dispatch/gating condition) must be scoped exactly to the cases being changed - a condition meant for a few quant types must not silently disable or enable everything else.
|
||||
- No hardcoded warp/lane size - use `ggml_cuda_get_physical_warp_size()` (32 on CUDA, 64 on HIP/ROCm) and the portable helpers.
|
||||
- Strip leftover debug/profiling/logging code before review.
|
||||
- New or changed op? Update `docs/ops.md` and the relevant `docs/ops/*.csv` for the touched backend.
|
||||
- New op or operator change needs corresponding `test-backend-ops` cases, and (per `CONTRIBUTING.md`) consistency across at least two backends.
|
||||
- New kernels are expected to come with concrete perf data (throughput across realistic tensor shapes), not just correctness.
|
||||
- Don't have a backend mutate the cgraph as a shortcut - that's an unresolved architectural question, not something to slip in.
|
||||
- Expect this to need two maintainer approvals; that's normal for `ggml/` changes, not a sign something is wrong.
|
||||
- For CUDA: Avoid excessively templating kernels, only add this where it shows visible performance gain.
|
||||
|
||||
## Public API (`include/llama.h`)
|
||||
|
||||
Public API changes carry a higher bar than internal ones (`CONTRIBUTING.md`). Review for:
|
||||
|
||||
- Justification: why doesn't an existing mechanism (e.g. `cb_eval`, existing batch/sampler knobs) suffice? If it does, the change likely shouldn't add public surface. This is the single most common reason these PRs are rejected.
|
||||
- Experimental or stop-gap surface belongs in a side header (`llama-ext.h`), not in `llama.h`.
|
||||
- Keep it minimal and general: prefer one general call over several narrow convenience wrappers; make new calls forward-compatible (e.g. mixed-modality batches) rather than assuming today's shape.
|
||||
- The C API is the first-class, stable, ABI-defining surface - don't propose a parallel C++ API as a replacement. `llama-cpp.h` stays a thin convenience layer.
|
||||
- Types and naming: sized integer types (`int32_t`, `size_t` for sizes/offsets); `snake_case`; `<class>_<method>` = `<class>_<action>_<noun>`; enum values upper-case and prefixed with the enum name; `_t` suffix for opaque types. Avoid gratuitous signature/ABI changes to existing exported functions.
|
||||
- Every new API needs a working example/tool exercising it in the same PR - reviewers find real bugs by requiring it to be wired into `server`, `embedding`, `perplexity`, etc.
|
||||
|
||||
## Server (`tools/server/`)
|
||||
|
||||
- Is the feature within server's defined scope? Check `tools/server/README-dev.md` - out-of-scope features get declined.
|
||||
- Security: don't trust client-supplied headers (e.g. `X-Forwarded-For`) or add footguns; things like IP allowlisting belong at a reverse proxy unless there's a trusted-proxy design.
|
||||
- Wire new behavior into the existing request/response and checkpoint paths correctly; watch for resource leaks across requests.
|
||||
|
||||
## WebUI (`tools/ui/`)
|
||||
|
||||
These apply only to changes under `tools/ui/` (formerly `tools/server/webui/`) - the SvelteKit + Svelte 5 frontend. They are frontend conventions, not repo-wide ones, so do not apply them to C++ or other tool code.
|
||||
|
||||
The dominant review theme in this area is placement: most accepted nits are about putting a value, function, or piece of logic in the layer where it belongs.
|
||||
|
||||
### Constants and enums - decision rules
|
||||
|
||||
Trigger: a literal that is reused, domain-significant, or crosses a boundary (wire/persisted format, selector, key, shared UI text). One-off component-local values stay inline (see the boundary at the end of this section) - flagging those is noise. Before adding, grep `$lib/constants` / `$lib/enums` - reuse beats a new near-duplicate. What becomes what:
|
||||
|
||||
- **String enum** (`$lib/enums`, `enum X { KEY = 'value' }`): a closed set of categorical values the code branches on, compares, or stores as a type discriminator - roles, statuses, kinds, types, actions, sort fields. Anything written as `x.type === 'some-string'` or a closed string union in a `.d.ts` is an enum. Keys are SCREAMING_SNAKE with a domain prefix (`ModelDownloadStopRequest.CANCEL`); values are lowercase by default and are the persisted/wire representation, so they may intentionally differ from the key (`TOOL_USE = 'tools'`, `MTP = 'mtp'`) or mirror an external API exactly - say so in the enum's JSDoc when they do (see `HfModelSort`). No numeric-valued enums.
|
||||
- **Constant object keyed by enum** (constants): a lookup/mapping - icon per file type, label per capability, HTTP code to message (`FILE_TYPE_ICONS`, `HTTP_CODE_TO_STRING`, `Record<Enum, string>`). The enum enumerates, the object maps.
|
||||
- **Declarative config array/object** (constants): a set of related items consumed by more than one renderer (e.g. dropdown + sheet) - menu items whose `action`/`id`/`enabledWhen` fields are enum values, not strings (`ATTACHMENT_FILE_ITEMS`). If two surfaces render the same menu, both must render from this one config.
|
||||
- **Grouped `as const` object** (constants): related literals that belong to one feature - error-message taxonomy (`ERROR_MESSAGES` with `HTTP`/`NETWORK` groups), UI copy shared across surfaces (`ATTACHMENT_MENU_TEXT`), a feature's prompt + fallback + patterns (`TITLE_GENERATION`). Prefer one grouped object over scattering same-feature constants.
|
||||
- **Scalar constants**:
|
||||
- strings: CSS class strings, selectors, `data-*` attribute names, header names, storage keys, API routes/paths, file extensions.
|
||||
- numbers: any threshold, timeout, size, limit, or count - suffixed with unit or context (`_MS`, `_PX`, `_BYTES`, `_CHARS`, `_MAX_LENGTH`).
|
||||
- regexes: any non-trivial literal regex, `_REGEX` suffix, grouped in the feature's constants file.
|
||||
- characters/separators: named by meaning in their domain, not by character - `PATH_SEPARATOR`, `ORG_SEPARATOR`, `SSE_RECORD_SEPARATOR`, `ISO_DATE_TIME_SEPARATOR`, `NEWLINE`. No universal `SLASH` constant.
|
||||
- **Key-name constants**: strings used as keys into another object or persisted config get an object constant mirroring the property names (`SETTINGS_KEYS.CUSTOM_CSS = 'customCss'`), so a rename breaks the build instead of silently missing.
|
||||
|
||||
Placement and composition:
|
||||
|
||||
- Shared across features -> `$lib/constants/<topic>.constants.ts` / `$lib/enums/<topic>.enums.ts`; feature-dedicated values may live in the feature folder, but only what is truly dedicated to it.
|
||||
- Compose, don't duplicate: constants may derive from other constants/enums (`SANDBOX_TOOL_NAME = BuiltInTool.BROWSER_RUN_JAVASCRIPT`, storage keys built from the `STORAGE_APP_NAME` prefix).
|
||||
- Tests and Storybook stories use the same constants/enums for structural assertions - they don't re-declare the literals they assert against. Exception: persisted/wire-contract assertions keep independent literals - there the literal value itself is the contract, and reusing the production constant would make the test tautological (change the constant, the expectation changes with it, the regression still passes).
|
||||
|
||||
What stays inline: one-off user-facing copy inside a single component, console/log messages, and import specifiers. When inline copy gains a second consumer, it graduates into a grouped constant object.
|
||||
|
||||
Adjacent data-shape rule: empty string is not a valid "unset" initial state - use `undefined`.
|
||||
|
||||
### Placement and layering
|
||||
|
||||
- Components render; they don't own business logic. Simple presentation-local derivation stays in the component as `$derived` from props and store values - what leaves is shared business logic, multi-store orchestration, and naming/parsing logic that a store method, `$lib/utils` function, or `$lib/hooks` hook should own. A component should call `store.doSomething()` rather than reimplement it.
|
||||
- Keep the layers deliberate: stores (`*.svelte.ts`, stateful), services (stateless clients), utils (pure functions), hooks (`*.svelte.ts`, reusable reactivity), `$lib/types` (types), `$lib/constants` (constants). If a new file blurs these, flag it and name the layer it belongs in.
|
||||
- A stateful "service" is misnamed - it's a store or belongs in the scope that uses it. Conversely, a one-off `.svelte.ts` module (which is effectively a store) for logic only one component uses is overkill; keep it in that component or a context.
|
||||
- Before introducing a new abstraction (store/service/hook/context), ask whether it will ever have a second consumer; if not, keep the logic in the existing scope. KISS first, iterate later.
|
||||
- Barrel files (`index.ts`) match the existing barrel style - file-level JSDoc plus per-export group comments for stores, plain re-export lists for constants/enums - and add JSDoc only where an export isn't self-explanatory; constants and types are not re-exported from store barrels.
|
||||
|
||||
### DRY and dead code
|
||||
|
||||
- Reuse before writing: grep for an existing helper, component, or store method before adding a new one (truncation, copy/download, file preview, sanitization, and formatting helpers all already exist).
|
||||
- Sibling variants (e.g. Dropdown vs Sheet versions of the same picker) must render from one shared config / single source of truth, not duplicate markup and logic.
|
||||
- Flag dead code, commented-out markup, and leftovers from renamed or removed features. Intentional-but-temporary leftovers get `@deprecated` JSDoc so they can be found and removed later.
|
||||
|
||||
### Components and Svelte 5
|
||||
|
||||
- Break bloated components into focused sub-components (plus dedicated `*Skeleton.svelte` loading states); use `{#snippet}` for repeated in-component markup.
|
||||
- Use the existing `$lib/components/ui` primitives instead of hand-rolling dialogs, tooltips, inputs, or dropdowns.
|
||||
- Rune syntax only: `$props()` deconstruction, `$derived` (reach for `$derived.by` only when a simple expression won't do), `$state`. Avoid `$effect` for things `$state`/`$derived` can express - a wall of `$effect` is a design smell.
|
||||
- Handle window/document events via `<svelte:window>` / `<svelte:document>`, not manual add/remove-listener plumbing.
|
||||
- Svelte 5 supports arrays in the `class` attribute - don't reach for `cn()` for simple conditional classes.
|
||||
- Use `SvelteMap`/`SvelteSet` for reactive collections.
|
||||
|
||||
### Comments
|
||||
|
||||
- Same bar as the General checklist, enforced harder here: no comments that restate the code, no verbose JSDoc on obvious functions, no commented-out code. Examples and edge-case enumerations live in unit tests, not prose; non-obvious why/rationale comments stay, because a test cannot carry that context at the code location.
|
||||
|
||||
### Types, naming, and style
|
||||
|
||||
- `===` over `==`, `includes()` over index checks, no redundant null-coalescing after an `Array.isArray` check.
|
||||
- One spelling per concept across the codebase (e.g. pick `cwd` vs `currentWorkingDirectory` and apply it everywhere); explicit, self-describing names over clever ones.
|
||||
|
||||
### Performance (streaming UI)
|
||||
|
||||
- Anything running per streamed chunk must be cheap: throttle before computing, not after; apply block-level regex passes globally and scan lines backwards with early exit; don't re-parse the whole message per update.
|
||||
- Smooth height/expand transitions consistent with the rest of the UI - animate auto-height via `interpolate-size: allow-keywords` with a `grid-template-rows: 0fr/1fr` fallback for browsers without it (see `CollapsibleRegion.svelte`).
|
||||
|
||||
### Security (frontend side)
|
||||
|
||||
- Sanitize anything rendered from model or remote data: the rehype/`{@html}` pipeline, SVG content, and URLs (use the `$lib/utils` URL sanitizer rather than ad-hoc checks or a third-party lib for trivial cases).
|
||||
- Proxy headers and agent/tool filesystem access are security boundaries - scope enforcement (e.g. browse-root sandboxing) must be validated end-to-end, not just in the UI; loop in server reviewers for the C++ side.
|
||||
|
||||
### PR hygiene specific to `tools/ui`
|
||||
|
||||
- Build artifacts are not committed - the UI is built by CI (`.github/workflows/ui-build.yml`) and downloaded/bundled at release time. `npm run build` must still succeed locally; a red UI build job is blocking.
|
||||
- `npm run lint && npm run check` (and `npm run format`) must pass; a failing Storybook test is blocking.
|
||||
- PRs with user-facing interactive or visual changes need a short video demo in the description showing the change working (screenshots are acceptable for pure-visual tweaks). Test-only, refactor, docs, and config PRs don't need a demo.
|
||||
- Scope discipline: changes that belong to another stacked PR don't ride along in this one - call them out and move them there. Flag unrelated edits to deprecated/legacy files and ask whether they're intentional.
|
||||
- New nontrivial utils and parsing logic get Vitest unit tests; reusable UI components get Storybook stories/tests when practical.
|
||||
|
||||
## Multimodal (`tools/mtmd/`)
|
||||
|
||||
- Tensor names must be prefixed by `v.`, `a.`, `mm.` or `a.mm.` (legacy naming doesn't follow this convention - this is expected, but new code should follow it).
|
||||
- Do not use explicit sin/cos for RoPE; use `ggml_rope_ext` instead, see `HOWTO-add-model.md`. If it can't express the needed behavior, that's a design discussion, not a PR.
|
||||
- New GGML ops must not be introduced in the same PR, you must push it as a separate PR.
|
||||
- In most cases, `build_vit` should be enough to build the transformer graph for vision models. Do not add a loop to build the transformer graph manually, unless you have a very good reason to do so. If you do, please explain why in the PR description.
|
||||
- If you need a dedicated preprocessor, there is a high chance that it can be a derived class from one of the existing preprocessors. Check carefully before adding a new preprocessor class.
|
||||
- If the model need a new public API in `mtmd.h`, open a discussion first.
|
||||
- For audio generation models, see `tools/mtmd/README-dev.md`
|
||||
|
||||
## Tests
|
||||
|
||||
- Follow the existing testing patterns. Do not add a new testing system. Before adding a new file under `tests/*`, think carefully about whether the tests can go into an existing file first.
|
||||
- In most cases, new test cases belong in an existing test file - check for one covering the same component before adding anything new.
|
||||
- Only add tests that bring meaningful results. Too-trivial tests just bloat the suite and CI.
|
||||
- No time-sensitive tests (timing thresholds, sleeps, races against wall-clock); they are flaky on CI.
|
||||
- Think twice about tests that significantly increase CI run time (expensive computation, large inputs, or long sleep/wait delays) or download large amounts of data from the internet (big models, datasets). Flag them and ask whether a smaller model/input or an existing fixture would do.
|
||||
- Server: small tests of internal components (helpers, `server_tokens`, queues, format converters, ...) belong in `tests/test-server-component.cpp`. The end-to-end pytest suite in `tools/server/tests` is costly in CI time; only add to it when the test requires end-to-end behavior with a real model.
|
||||
|
||||
## General (always)
|
||||
|
||||
Enforce the `AGENTS.md` / `CONTRIBUTING.md` coding and naming guidelines on every changed line - this is a distinct pass from checking that the code works, and matters just as much for review speed:
|
||||
|
||||
- ASCII only in code and comments - no emdash, unicode arrows, `x`, `...` used as unicode; use `-`, `->`, `x`, `...` ASCII equivalents.
|
||||
- Comments are concise and explain non-obvious *why*, not *what*. Flag verbose comments, comments that restate the code, comments that reference the current task/PR, and comments hard-wrapped to a fixed column width.
|
||||
- Do not force-wrap prose/comments to a fixed character count or split a sentence across lines.
|
||||
- `snake_case` names; `kebab-case` (lowercase-with-dashes) file names for C/C++, `.h` headers; Python files lowercase-with-underscores. Naming optimizes for longest common prefix (`number_small`, not `small_number`).
|
||||
- 4-space indentation, brackets on the same line, `void * ptr`, `int & a`, no trailing whitespace; match the surrounding style.
|
||||
- Before pushing the PR, run the code style check and editorconfig check locally (see `.github/workflows/code-style.yml` and `.github/workflows/editorconfig.yml`). If the change touches Python code, also run the Python type check (`.github/workflows/python-type-check.yml`).
|
||||
- Reuse existing infrastructure over introducing new components; no new third-party dependencies, extra headers, or files unless clearly justified.
|
||||
- Keep it simple: a simpler change doing 90% is often preferable to a complex one doing 100%. Flag unnecessary templates/fancy STL; basic `for` loops are fine here.
|
||||
- Every added line should be something the contributor can explain and defend to a reviewer without AI help - flag anything that looks copied-in without understanding.
|
||||
- `Co-authored-by:` must be reserved for human co-authors; AI contributions (claude, cursor, codex, etc.) must use `Assisted-by:`; if this point is violated, it's a blocking finding.
|
||||
- Any mentions of Minja must be treated as blocking; see `AGENTS.md` for why.
|
||||
|
||||
## Reporting
|
||||
|
||||
Group findings by severity so the user knows what actually blocks a merge:
|
||||
|
||||
1. **Blocking** - quick-reject/scope issues and correctness bugs; these can sink the PR regardless of everything else.
|
||||
2. **Will slow the review** - convention/naming/comment violations, missing tests/docs/perf data, missing API justification or example.
|
||||
3. **Nits** - minor style, optional cleanups.
|
||||
|
||||
For each finding, point to the file and line and say concretely what to change and why. Do not rewrite the whole diff unprompted; let the contributor make the fixes so they own and understand them. And do not draft any PR text, commit message, or reviewer reply - that is the contributor's to write.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: slice-model
|
||||
description: Build a small test "slice" of a HF model (fewer layers/experts) by downloading only the needed tensors via HTTP range requests. Use when the user needs a small version of a big model to test conversion or inference.
|
||||
---
|
||||
|
||||
# Slice a HF model for testing
|
||||
|
||||
[slice_model.py](slice_model.py) is a **reference** implementation. Read it first: its docstring and `--help` explain how it works and what it does. Its heuristics only cover common layouts, and new models often have tensor names, shapes or config keys it does not handle. Do not run it blindly. If the model needs special handling, copy it to `tmp/slice_<model>.py` and adapt the copy, or write a new script with the same approach.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Ask the user for the repo, what the slice must exercise (e.g. a MoE layer, each layer type of a hybrid model, MTP, the vision encoder), the size budget and the output dir.
|
||||
2. Inspect the model: run the script with `--inspect`, read `config.json`, and read the HF modeling code if anything is unclear. Check the plan against the pitfalls below.
|
||||
3. Build the slice, then read the config patches in the log and check each one.
|
||||
4. Verify: load the slice in transformers (no missing/unexpected weight warnings, no NaN logits), then convert it with `convert_hf_to_gguf.py` and run it. For numerical checks, see `examples/model-conversion/`.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- Layer counts in nested or unknown config keys, or several towers with the same layer count.
|
||||
- Hybrid layer patterns (`layer_types`, `full_attention_interval`, ...): keep at least one layer of each type, and check the patched config still matches the kept layers.
|
||||
- Dense leading layers (`first_k_dense_replace`) and MTP/nextn layers numbered after the regular layers.
|
||||
- Every per-expert tensor must be sliced to the same expert count: experts, router weight and biases. Tensors with the expert dim not at dim 0 are not contiguous.
|
||||
- Routing constraints: top-k, expert groups, shared experts, zero-compute experts.
|
||||
- Quantized weights (FP8, MXFP4): slice the scale tensors in the same way as their weight.
|
||||
- Sizes derived in modeling code instead of stored in the config: follow the exact formula if you shrink them.
|
||||
- Slicing the vocab without the tokenizer breaks conversion.
|
||||
@@ -0,0 +1,560 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
REFERENCE ONLY - build a small test "slice" of a HF safetensors model without downloading the full weights.
|
||||
|
||||
The heuristics below (layer/expert detection, config patching) cover common layouts only. New models often use tensor names, shapes or config keys that this script does not know about. Always run --inspect first, check the plan, and adapt the script to the model.
|
||||
|
||||
How it works: each safetensors shard starts with an 8-byte header length + JSON header, which gives the exact byte span of every tensor. We fetch only the headers, then use HTTP range requests to fetch the tensors we keep:
|
||||
1. keep only some layers (per numeric "family" in tensor names whose count matches a layer count in config.json), renumbered to 0..n-1
|
||||
2. keep only the first E experts (per-expert tensors are dropped, stacked expert tensors and router weights are row-sliced along dim 0)
|
||||
3. optionally keep only the first V rows of vocab-sized tensors (off by default, see --vocab-size)
|
||||
config.json is patched to match. Other small repo files are copied as-is.
|
||||
|
||||
Usage:
|
||||
python slice_model.py Qwen/Qwen3-0.6B --inspect
|
||||
python slice_model.py Qwen/Qwen3-30B-A3B --num-layers 2 --num-experts 8 --dry-run
|
||||
python slice_model.py Qwen/Qwen3-30B-A3B --keep-layers 0,3 --output ../models/qwen3moe-slice
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import concurrent.futures as cf
|
||||
import fnmatch
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import requests
|
||||
from huggingface_hub import HfApi, get_token, hf_hub_url
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
logger = logging.getLogger("slice_model")
|
||||
|
||||
INDEX_FILE = "model.safetensors.index.json"
|
||||
CONFIG_FILE = "config.json"
|
||||
|
||||
# files not worth copying into a slice
|
||||
AUX_BLACKLIST = [
|
||||
# weights in other formats (the slice replaces them)
|
||||
"*.bin", "*.pth", "*.pt", "*.ckpt", "*.h5", "*.msgpack",
|
||||
"*.onnx", "*.tflite", "*.mlmodel", "*.mlpackage/*", "*.gguf", "*.nemo",
|
||||
"onnx/*", "openvino/*", "coreml/*", "original/*", "metal/*",
|
||||
# media / demo assets
|
||||
"*.png", "*.jpg", "*.jpeg", "*.gif", "*.webp", "*.avif", "*.svg",
|
||||
"*.mp4", "*.webm", "*.mov", "*.wav", "*.mp3", "*.flac", "*.ogg", "*.pdf",
|
||||
".gitattributes",
|
||||
]
|
||||
AUX_MAX_SIZE = 50 * 1024 * 1024
|
||||
|
||||
LAYER_CONTAINERS = {"layers", "layer", "blocks", "block", "h"}
|
||||
EXPERT_ROW_KEYWORDS = ("expert", "router", "gate", "score")
|
||||
|
||||
LAYER_KEYS = {
|
||||
"num_hidden_layers", "num_layers", "n_layer", "n_layers", "num_layer",
|
||||
"num_decoder_layers", "encoder_layers", "decoder_layers", "depth",
|
||||
"num_encoder_layers", "num_attention_layers", "layer_count",
|
||||
}
|
||||
VOCAB_KEYS = {"vocab_size", "padded_vocab_size"}
|
||||
NEXTN_KEYS = {"num_nextn_predict_layers", "num_mtp_layers"}
|
||||
TOPK_SUBSTR = ("per_tok", "topk", "top_k", "moe_k")
|
||||
EXPERT_KEY_EXCLUDE = ("per_tok", "topk", "top_k", "shared", "group", "intermediate", "size", "dim", "dtype")
|
||||
|
||||
|
||||
def parse_size(text: str) -> int:
|
||||
m = re.fullmatch(r"(\d+(?:\.\d+)?)\s*([kKmMgGtT]?)[bB]?", text.strip())
|
||||
if not m:
|
||||
raise argparse.ArgumentTypeError(f"invalid size: {text!r}")
|
||||
mult = {"": 1, "k": 1024, "m": 1024**2, "g": 1024**3, "t": 1024**4}[m.group(2).lower()]
|
||||
return int(float(m.group(1)) * mult)
|
||||
|
||||
|
||||
def human(n: float) -> str:
|
||||
for unit in ("B", "KB", "MB", "GB"):
|
||||
if n < 1024 or unit == "GB":
|
||||
return f"{n:.1f}{unit}" if unit != "B" else f"{int(n)}B"
|
||||
n /= 1024
|
||||
return f"{n}GB"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------
|
||||
# HTTP
|
||||
# --------------------------------------------------------------------------------------
|
||||
|
||||
_tls = threading.local()
|
||||
|
||||
|
||||
def _session() -> requests.Session:
|
||||
s = getattr(_tls, "s", None)
|
||||
if s is None:
|
||||
s = requests.Session()
|
||||
_tls.s = s
|
||||
return s
|
||||
|
||||
|
||||
class HubClient:
|
||||
def __init__(self, repo_id: str, revision: str, token: str | None):
|
||||
self.repo_id = repo_id
|
||||
self.revision = revision
|
||||
self.headers = {"Authorization": f"Bearer {token}"} if token else {}
|
||||
|
||||
def url(self, filename: str) -> str:
|
||||
return hf_hub_url(self.repo_id, filename, revision=self.revision)
|
||||
|
||||
def _request(self, filename: str, headers: dict, stream: bool = False, retries: int = 5):
|
||||
last: Exception | None = None
|
||||
for attempt in range(1, retries + 1):
|
||||
try:
|
||||
r = _session().get(self.url(filename), headers=headers, stream=stream, timeout=90, allow_redirects=True)
|
||||
if r.status_code in (429, 500, 502, 503, 504):
|
||||
raise requests.HTTPError(f"HTTP {r.status_code}", response=r)
|
||||
r.raise_for_status()
|
||||
return r
|
||||
except Exception as e: # noqa: BLE001
|
||||
last = e
|
||||
if attempt == retries or "404" in str(e) or "401" in str(e) or "403" in str(e):
|
||||
break
|
||||
time.sleep(2.0 * attempt)
|
||||
raise RuntimeError(f"{self.repo_id}/{filename}: {last}") from last
|
||||
|
||||
def get_range(self, filename: str, start: int, end_inclusive: int) -> bytes:
|
||||
h = dict(self.headers)
|
||||
h["Range"] = f"bytes={start}-{end_inclusive}"
|
||||
return self._request(filename, h).content
|
||||
|
||||
def get_json(self, filename: str) -> Any:
|
||||
return json.loads(self._request(filename, dict(self.headers)).content)
|
||||
|
||||
def download_file(self, filename: str, dest: Path):
|
||||
r = self._request(filename, dict(self.headers), stream=True)
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(dest, "wb") as f:
|
||||
for chunk in r.iter_content(chunk_size=4 * 1024 * 1024):
|
||||
if chunk:
|
||||
f.write(chunk)
|
||||
|
||||
def stream_range_into_fd(self, filename: str, start: int, end_inclusive: int, fd: int, write_offset: int):
|
||||
h = dict(self.headers)
|
||||
h["Range"] = f"bytes={start}-{end_inclusive}"
|
||||
r = self._request(filename, h, stream=True)
|
||||
pos = write_offset
|
||||
for chunk in r.iter_content(chunk_size=8 * 1024 * 1024):
|
||||
if chunk:
|
||||
os.pwrite(fd, chunk, pos)
|
||||
pos += len(chunk)
|
||||
if pos - write_offset != end_inclusive - start + 1:
|
||||
raise RuntimeError(f"short read: {filename} bytes={start}-{end_inclusive}")
|
||||
|
||||
|
||||
def fetch_shard_header(client: HubClient, filename: str) -> tuple[dict, int]:
|
||||
(header_len,) = struct.unpack("<Q", client.get_range(filename, 0, 7))
|
||||
header = json.loads(client.get_range(filename, 8, 8 + header_len - 1))
|
||||
header.pop("__metadata__", None)
|
||||
return header, 8 + header_len
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------
|
||||
# config.json analysis / patching
|
||||
# --------------------------------------------------------------------------------------
|
||||
|
||||
def walk_config(cfg: Any):
|
||||
"""Yield (container_dict, key, value) for every leaf in nested dicts."""
|
||||
if isinstance(cfg, dict):
|
||||
for k, v in cfg.items():
|
||||
if isinstance(v, dict):
|
||||
yield from walk_config(v)
|
||||
else:
|
||||
yield cfg, k, v
|
||||
|
||||
|
||||
@dataclass
|
||||
class ConfigInfo:
|
||||
cfg: dict | None
|
||||
layer_counts: set[int] = field(default_factory=set)
|
||||
nextn_counts: set[int] = field(default_factory=set)
|
||||
expert_counts: set[int] = field(default_factory=set)
|
||||
vocab_sizes: set[int] = field(default_factory=set)
|
||||
first_k_dense: int = 0
|
||||
|
||||
|
||||
def analyze_config(cfg: dict | None) -> ConfigInfo:
|
||||
info = ConfigInfo(cfg=cfg)
|
||||
if cfg is None:
|
||||
return info
|
||||
for _d, key, val in walk_config(cfg):
|
||||
if not isinstance(val, int) or isinstance(val, bool) or val <= 0:
|
||||
continue
|
||||
if key in LAYER_KEYS:
|
||||
info.layer_counts.add(val)
|
||||
elif key in NEXTN_KEYS:
|
||||
info.nextn_counts.add(val)
|
||||
elif key in VOCAB_KEYS:
|
||||
info.vocab_sizes.add(val)
|
||||
elif key == "first_k_dense_replace":
|
||||
info.first_k_dense = val
|
||||
elif "expert" in key and val > 1 and not any(x in key for x in EXPERT_KEY_EXCLUDE):
|
||||
info.expert_counts.add(val)
|
||||
return info
|
||||
|
||||
|
||||
def patch_config(cfg: dict, families: dict[str, Family], new_vocab: int | None, orig_vocabs: set[int]) -> list[str]:
|
||||
"""Patch counts in-place to match the slice. Returns log lines."""
|
||||
log = []
|
||||
by_card: dict[int, Family] = {}
|
||||
for fam in families.values():
|
||||
if fam.kept is not None:
|
||||
by_card.setdefault(fam.card, fam)
|
||||
by_card.setdefault(fam.regular_total, fam)
|
||||
max_new_experts = max((f.new_count for f in families.values() if f.kind == "expert" and f.kept is not None), default=None)
|
||||
|
||||
for d, key, val in list(walk_config(cfg)):
|
||||
if isinstance(val, int) and not isinstance(val, bool):
|
||||
fam = by_card.get(val)
|
||||
if key in LAYER_KEYS and fam is not None and fam.kind == "layer":
|
||||
d[key] = fam.new_regular
|
||||
elif key == "first_k_dense_replace":
|
||||
lfam = next((f for f in families.values() if f.kind == "layer"), None)
|
||||
if lfam is None:
|
||||
continue
|
||||
assert lfam.kept is not None
|
||||
d[key] = sum(1 for i in lfam.kept if i < val)
|
||||
elif "expert" in key and fam is not None and fam.kind == "expert" and not any(x in key for x in EXPERT_KEY_EXCLUDE):
|
||||
d[key] = fam.new_count
|
||||
elif "expert" in key and any(x in key for x in TOPK_SUBSTR) and max_new_experts is not None and val > max_new_experts:
|
||||
d[key] = max_new_experts
|
||||
elif key in VOCAB_KEYS and new_vocab is not None and val in orig_vocabs:
|
||||
d[key] = new_vocab
|
||||
if d[key] != val:
|
||||
log.append(f"{key}: {val} -> {d[key]}")
|
||||
elif isinstance(val, list) and val and all(not isinstance(x, (dict, list)) for x in val):
|
||||
# per-layer lists, e.g. layer_types
|
||||
fam = by_card.get(len(val))
|
||||
if fam is not None and fam.kind == "layer" and len(val) in (fam.card, fam.regular_total):
|
||||
assert fam.kept is not None
|
||||
d[key] = [val[i] for i in fam.kept if i < len(val)]
|
||||
log.append(f"{key}: list[{len(val)}] -> list[{len(d[key])}]")
|
||||
return log
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------
|
||||
# tensor name analysis
|
||||
# --------------------------------------------------------------------------------------
|
||||
|
||||
@dataclass
|
||||
class Family:
|
||||
"""One numeric slot in tensor names, e.g. 'model.layers.#'."""
|
||||
prefix: str
|
||||
kind: str # "layer" | "expert" | "other"
|
||||
indices: set[int] = field(default_factory=set)
|
||||
kept: list[int] | None = None # None = keep all, no renumbering
|
||||
renumber: dict[int, int] = field(default_factory=dict)
|
||||
regular_total: int = 0 # layer count excluding MTP tail
|
||||
new_regular: int = 0
|
||||
|
||||
@property
|
||||
def card(self) -> int:
|
||||
return len(self.indices)
|
||||
|
||||
@property
|
||||
def new_count(self) -> int:
|
||||
return len(self.kept) if self.kept is not None else self.card
|
||||
|
||||
|
||||
def name_slots(name: str):
|
||||
"""Yield (slot_pos, family_prefix, index) for each numeric path component."""
|
||||
tokens = name.split(".")
|
||||
norm: list[str] = []
|
||||
for i, tok in enumerate(tokens):
|
||||
if tok.isdigit():
|
||||
yield i, ".".join(norm), int(tok)
|
||||
norm.append("#")
|
||||
else:
|
||||
norm.append(tok)
|
||||
|
||||
|
||||
def build_families(names: list[str], cfg_info: ConfigInfo) -> dict[str, Family]:
|
||||
families: dict[str, Family] = {}
|
||||
for name in names:
|
||||
tokens = name.split(".")
|
||||
for i, prefix, idx in name_slots(name):
|
||||
container = tokens[i - 1] if i > 0 else ""
|
||||
fam = families.get(prefix)
|
||||
if fam is None:
|
||||
fam = Family(prefix=prefix, kind="expert" if "expert" in container else "other")
|
||||
families[prefix] = fam
|
||||
fam.indices.add(idx)
|
||||
|
||||
acceptable_layer_cards: dict[int, int] = {} # card -> regular layer count
|
||||
for c in cfg_info.layer_counts:
|
||||
acceptable_layer_cards[c] = c
|
||||
for m in cfg_info.nextn_counts:
|
||||
acceptable_layer_cards[c + m] = c
|
||||
|
||||
for fam in families.values():
|
||||
if fam.kind == "expert":
|
||||
continue
|
||||
container = fam.prefix.rsplit(".", 1)[-1]
|
||||
if cfg_info.cfg is not None:
|
||||
if fam.card in acceptable_layer_cards:
|
||||
fam.kind = "layer"
|
||||
fam.regular_total = acceptable_layer_cards[fam.card]
|
||||
elif fam.card >= 4 and container in LAYER_CONTAINERS:
|
||||
fam.kind = "layer"
|
||||
fam.regular_total = fam.card
|
||||
return families
|
||||
|
||||
|
||||
def plan_families(families: dict[str, Family], num_layers: int, keep_layers: list[int] | None, num_experts: int, first_k_dense: int):
|
||||
for fam in families.values():
|
||||
if fam.kind == "layer":
|
||||
if keep_layers is not None:
|
||||
kept = [i for i in keep_layers if i < fam.regular_total]
|
||||
else:
|
||||
base = min(num_layers, fam.regular_total)
|
||||
kept = list(range(base))
|
||||
# keep one MoE layer for models whose first k layers are dense
|
||||
if 0 < first_k_dense < fam.regular_total and first_k_dense >= base >= 1:
|
||||
kept = list(range(base - 1)) + [first_k_dense]
|
||||
fam.new_regular = len(kept)
|
||||
# keep the MTP tail (layer indices beyond the regular count)
|
||||
kept += [i for i in sorted(fam.indices) if i >= fam.regular_total]
|
||||
fam.kept = sorted(set(kept))
|
||||
elif fam.kind == "expert":
|
||||
fam.kept = sorted(fam.indices)[:num_experts]
|
||||
fam.new_regular = len(fam.kept)
|
||||
else:
|
||||
fam.kept = None
|
||||
continue
|
||||
fam.renumber = {old: new for new, old in enumerate(fam.kept)}
|
||||
|
||||
|
||||
def slice_tensor_name(name: str, families: dict[str, Family]) -> str | None:
|
||||
"""Return the renumbered output name, or None if the tensor is dropped."""
|
||||
tokens = name.split(".")
|
||||
for i, prefix, idx in name_slots(name):
|
||||
fam = families[prefix]
|
||||
if fam.kept is None:
|
||||
continue
|
||||
if idx not in fam.renumber:
|
||||
return None
|
||||
tokens[i] = str(fam.renumber[idx])
|
||||
return ".".join(tokens)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------
|
||||
# slicing
|
||||
# --------------------------------------------------------------------------------------
|
||||
|
||||
@dataclass
|
||||
class TensorPlan:
|
||||
src_name: str
|
||||
out_name: str
|
||||
dtype: str
|
||||
shape: list[int]
|
||||
orig_shape: list[int]
|
||||
size: int
|
||||
src_file: str
|
||||
seg_start: int # absolute byte range in src_file
|
||||
seg_end: int # inclusive
|
||||
|
||||
|
||||
def pick_weight_files(filenames: list[str]) -> tuple[list[str], bool]:
|
||||
"""Return (weight files, has_index). Preference: index -> model.safetensors -> any root safetensors."""
|
||||
if INDEX_FILE in filenames:
|
||||
return [], True
|
||||
if "model.safetensors" in filenames:
|
||||
return ["model.safetensors"], False
|
||||
root_st = [f for f in filenames if f.endswith(".safetensors") and "/" not in f]
|
||||
if "consolidated.safetensors" in root_st:
|
||||
return ["consolidated.safetensors"], False
|
||||
return root_st[:1], False
|
||||
|
||||
|
||||
def row_slice(shape: list[int], size: int, keep_rows: int) -> tuple[list[int], int] | None:
|
||||
# only a prefix of dim 0 is a single contiguous byte range
|
||||
if len(shape) < 1 or shape[0] <= keep_rows or size % shape[0] != 0:
|
||||
return None
|
||||
row_bytes = size // shape[0]
|
||||
return [keep_rows] + shape[1:], keep_rows * row_bytes
|
||||
|
||||
|
||||
def build_plan(args, families, headers, weight_files, src_file_of, cfg_info) -> list[TensorPlan]:
|
||||
plan_families(families, args.num_layers, args.keep_layers, args.num_experts, cfg_info.first_k_dense)
|
||||
plans = []
|
||||
for fname in weight_files:
|
||||
header, data_start = headers[fname]
|
||||
for name, meta in header.items():
|
||||
out_name = slice_tensor_name(name, families)
|
||||
if out_name is None:
|
||||
continue
|
||||
start, end = meta["data_offsets"]
|
||||
shape, size = list(meta["shape"]), end - start
|
||||
new_rows = None
|
||||
if args.vocab_size and shape and shape[0] > args.vocab_size \
|
||||
and any(shape[0] >= vs and shape[0] - vs <= 1024 for vs in cfg_info.vocab_sizes):
|
||||
new_rows = args.vocab_size
|
||||
elif shape and shape[0] in cfg_info.expert_counts and shape[0] > args.num_experts \
|
||||
and any(k in name for k in EXPERT_ROW_KEYWORDS):
|
||||
new_rows = args.num_experts
|
||||
if new_rows is not None:
|
||||
sliced = row_slice(shape, size, new_rows)
|
||||
if sliced is not None:
|
||||
shape, size = sliced
|
||||
plans.append(TensorPlan(name, out_name, meta["dtype"], shape, list(meta["shape"]), size, src_file_of[name],
|
||||
data_start + start, data_start + start + size - 1))
|
||||
return plans
|
||||
|
||||
|
||||
def print_inspect(cfg_info: ConfigInfo, families: dict[str, Family], plans: list[TensorPlan], all_names: list[str]):
|
||||
logger.info("=== config analysis ===")
|
||||
logger.info(f"layer counts: {sorted(cfg_info.layer_counts)} nextn: {sorted(cfg_info.nextn_counts)} "
|
||||
f"experts: {sorted(cfg_info.expert_counts)} vocab: {sorted(cfg_info.vocab_sizes)} first_k_dense: {cfg_info.first_k_dense}")
|
||||
logger.info("\n=== numeric families in tensor names ===")
|
||||
for fam in sorted(families.values(), key=lambda f: f.prefix):
|
||||
kept = "all" if fam.kept is None else f"{fam.kept[:16]}{'...' if len(fam.kept) > 16 else ''}"
|
||||
logger.info(f"{fam.prefix:60s} kind={fam.kind:6s} count={fam.card:4d} kept={kept}")
|
||||
logger.info("\n=== kept tensors (first index of each family only) ===")
|
||||
seen = set()
|
||||
for t in plans:
|
||||
key = re.sub(r"\.\d+\.", ".#.", t.out_name)
|
||||
if key in seen and t.shape == t.orig_shape:
|
||||
continue
|
||||
seen.add(key)
|
||||
mark = f" (sliced from {t.orig_shape})" if t.shape != t.orig_shape else ""
|
||||
logger.info(f"{t.out_name:70s} {t.dtype:5s} {t.shape}{mark}")
|
||||
kept_src = {t.src_name for t in plans}
|
||||
dropped = [n for n in all_names if n not in kept_src]
|
||||
logger.info(f"\n{len(plans)}/{len(all_names)} tensors kept, {len(dropped)} dropped")
|
||||
|
||||
|
||||
def write_slice(out_dir: Path, client: HubClient, plans: list[TensorPlan], jobs: int):
|
||||
header_out: dict[str, Any] = {}
|
||||
offset = 0
|
||||
for t in plans:
|
||||
header_out[t.out_name] = {"dtype": t.dtype, "shape": t.shape, "data_offsets": [offset, offset + t.size]}
|
||||
offset += t.size
|
||||
header_out["__metadata__"] = {"format": "pt"}
|
||||
hbytes = json.dumps(header_out, separators=(",", ":")).encode()
|
||||
hbytes += b" " * ((-(len(hbytes) + 8)) % 8)
|
||||
data_start = 8 + len(hbytes)
|
||||
|
||||
shard_path = out_dir / "model.safetensors"
|
||||
with open(shard_path, "wb") as f:
|
||||
f.write(struct.pack("<Q", len(hbytes)))
|
||||
f.write(hbytes)
|
||||
f.truncate(data_start + offset)
|
||||
fd = os.open(shard_path, os.O_WRONLY)
|
||||
try:
|
||||
def fetch(t: TensorPlan):
|
||||
dst = data_start + header_out[t.out_name]["data_offsets"][0]
|
||||
client.stream_range_into_fd(t.src_file, t.seg_start, t.seg_end, fd, dst)
|
||||
|
||||
with cf.ThreadPoolExecutor(max_workers=jobs) as ex:
|
||||
list(ex.map(fetch, plans))
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("repo", help="HF repo id, e.g. Qwen/Qwen3-0.6B")
|
||||
ap.add_argument("--revision", default=None)
|
||||
ap.add_argument("--output", type=Path, default=None, help="output dir (default: ./tmp/hf_slices/{user}--{model})")
|
||||
ap.add_argument("--inspect", action="store_true", help="print config analysis, tensor families and the plan, write nothing")
|
||||
ap.add_argument("--dry-run", action="store_true", help="plan and estimate size only, write nothing")
|
||||
ap.add_argument("--num-layers", type=int, default=2, help="keep the first N layers (default: 2)")
|
||||
ap.add_argument("--keep-layers", default=None, help="explicit comma-separated layer indices to keep, overrides --num-layers")
|
||||
ap.add_argument("--num-experts", type=int, default=8)
|
||||
ap.add_argument("--vocab-size", type=int, default=0,
|
||||
help="row-slice vocab-sized tensors to this size (default: 0 = off; tokenizer is NOT sliced, so this breaks conversion for most tokenizers)")
|
||||
ap.add_argument("--max-size", type=parse_size, default="1G", help="abort if the sliced weights exceed this (default: 1G)")
|
||||
ap.add_argument("--hf-token", default=None, help="HF token (default: HF_TOKEN env var or 'hf auth login' token)")
|
||||
ap.add_argument("--jobs", type=int, default=8, help="concurrent HTTP requests")
|
||||
args = ap.parse_args()
|
||||
args.keep_layers = sorted({int(x) for x in args.keep_layers.split(",")}) if args.keep_layers else None
|
||||
|
||||
token = args.hf_token or get_token()
|
||||
api = HfApi(token=token)
|
||||
info = api.model_info(args.repo, revision=args.revision, files_metadata=True)
|
||||
client = HubClient(args.repo, info.sha or "main", token)
|
||||
siblings = {s.rfilename: (s.size or 0) for s in info.siblings or []}
|
||||
filenames = sorted(siblings)
|
||||
|
||||
weight_files, has_index = pick_weight_files(filenames)
|
||||
weight_map: dict[str, str] = {}
|
||||
if has_index:
|
||||
weight_map = client.get_json(INDEX_FILE)["weight_map"]
|
||||
weight_files = sorted(set(weight_map.values()))
|
||||
elif not weight_files:
|
||||
logger.error("no .safetensors found")
|
||||
return 1
|
||||
|
||||
cfg = client.get_json(CONFIG_FILE) if CONFIG_FILE in filenames else None
|
||||
cfg_info = analyze_config(cfg)
|
||||
|
||||
with cf.ThreadPoolExecutor(max_workers=args.jobs) as ex:
|
||||
headers = dict(zip(weight_files, ex.map(lambda f: fetch_shard_header(client, f), weight_files)))
|
||||
all_names = [n for fname in weight_files for n in headers[fname][0]]
|
||||
src_file_of = weight_map if has_index else {n: weight_files[0] for n in all_names}
|
||||
|
||||
families = build_families(all_names, cfg_info)
|
||||
plans = build_plan(args, families, headers, weight_files, src_file_of, cfg_info)
|
||||
est = sum(t.size for t in plans)
|
||||
|
||||
if args.inspect:
|
||||
print_inspect(cfg_info, families, plans, all_names)
|
||||
logger.info(f"{args.repo}@{info.sha}: {len(plans)}/{len(all_names)} tensors, {human(est)}")
|
||||
if not any(f.kind == "layer" for f in families.values()):
|
||||
logger.warning("no layer family detected - check --inspect output and adapt the script")
|
||||
if est > args.max_size:
|
||||
logger.error(f"slice is {human(est)} > --max-size {human(args.max_size)}")
|
||||
return 1
|
||||
if args.inspect or args.dry_run:
|
||||
return 0
|
||||
|
||||
out_dir = args.output or Path("tmp") / "hf_slices" / args.repo.replace("/", "--")
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
write_slice(out_dir, client, plans, args.jobs)
|
||||
|
||||
skipped_aux = []
|
||||
for fname in filenames:
|
||||
if fname in weight_files or fname in (INDEX_FILE, CONFIG_FILE) or fname.endswith(".safetensors"):
|
||||
continue
|
||||
if any(fnmatch.fnmatch(fname, p) or fnmatch.fnmatch(Path(fname).name, p) for p in AUX_BLACKLIST):
|
||||
continue
|
||||
if siblings[fname] > AUX_MAX_SIZE:
|
||||
skipped_aux.append(f"{fname} ({human(siblings[fname])})")
|
||||
continue
|
||||
client.download_file(fname, out_dir / fname)
|
||||
for f in skipped_aux:
|
||||
logger.warning(f"skipped large aux file {f}")
|
||||
|
||||
patch_log = []
|
||||
if cfg is not None:
|
||||
new_vocab = args.vocab_size if args.vocab_size and any(vs > args.vocab_size for vs in cfg_info.vocab_sizes) else None
|
||||
patch_log = patch_config(cfg, families, new_vocab, cfg_info.vocab_sizes)
|
||||
with open(out_dir / CONFIG_FILE, "w") as f:
|
||||
json.dump(cfg, f, indent=2)
|
||||
f.write("\n")
|
||||
for line in patch_log:
|
||||
logger.info(f"config: {line}")
|
||||
|
||||
with open(out_dir / ".slice_meta.json", "w") as f:
|
||||
json.dump({
|
||||
"repo": args.repo, "revision": info.sha,
|
||||
"tensors": len(plans), "total_tensors": len(all_names), "size": est,
|
||||
"config_patches": patch_log, "skipped_aux": skipped_aux,
|
||||
}, f, indent=2)
|
||||
logger.info(f"wrote {out_dir}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
name: update-fusion-table
|
||||
description: Regenerate a per-device test-fusion baseline (tests/fusion/MTL.csv, ...) after fusion patterns or the dummy-model set change. Use when test-fusion --check fails.
|
||||
---
|
||||
|
||||
# Update a fusion baseline
|
||||
|
||||
`tests/fusion/<BACKEND>.csv` is the `test-fusion` golden baseline: `arch,moe,mode,label,count`,
|
||||
where `count` is how many times a fusion label matched per graph run. See `tests/fusion/README.md`.
|
||||
|
||||
Re-record only when the counts moved for the reason the contributor intended. A row failing on
|
||||
`nmse_fus` (> 1e-4) is a fused-vs-unfused correctness bug, not baseline drift - fix it, never
|
||||
record it away.
|
||||
|
||||
## Steps
|
||||
|
||||
```sh
|
||||
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON # enable the target backend
|
||||
cmake --build build --config Release --target test-llama-archs --target test-fusion -j
|
||||
|
||||
rm -rf build-ci-models && mkdir -p build-ci-models # always regenerate
|
||||
./build/bin/test-llama-archs -o build-ci-models
|
||||
|
||||
D=MTL0; B=tests/fusion/MTL.csv; T=/tmp/$(basename $B) # keep the target backend/device in sync
|
||||
./build/bin/test-fusion --models build-ci-models --device $D --check $B # read the FAIL rows
|
||||
./build/bin/test-fusion --models build-ci-models --device $D --record $T
|
||||
diff $B $T
|
||||
cp $T $B
|
||||
./build/bin/test-fusion --models build-ci-models --device $D --check $B # 0 failed, exit 0
|
||||
```
|
||||
|
||||
- Read the diff before applying it. Every added, removed or changed row must be explainable by the
|
||||
change under review; an unexplained row means the run picked up something unintended.
|
||||
- `expected 0` in a FAIL row = no baseline row for that label (new arch, or a fusion newly
|
||||
firing); non-zero `expected` = count changed; `fusion ... is no longer matched` = row stopped
|
||||
firing.
|
||||
- `--record` rewrites the whole file, so never record a partial run (`--model FILE`, a filtered
|
||||
models dir) - it drops the rows of every architecture that was not run.
|
||||
- `--record` marks every row `ok`; only the `--check` run validates.
|
||||
Reference in new issue
Block a user