zefie/nvfp4_gguf_quant
1
0 Fork 0
Code Actions Activity

include llama.cpp since there were some edits

Browse Source
This commit is contained in:
zefie committed 2026-11-01 16:32:35 -05:00
1 parent db7f28d087
commit 8c66990853
3918 files changed
+1640170 -1
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

llama.cpp/examples/llama-eval/README.md
+290
View File
@@ -0,0 +1,290 @@
# llama-eval
Simple evaluation tool for llama.cpp with support for multiple datasets.
For a full description, usage examples, and sample results, see:
- [PR 21152](https://github.com/ggml-org/llama.cpp/pull/21152)
## Quick start
```bash
# Single server
python3 llama-eval.py \
--server http://localhost:8033 \
--model my-model \
--dataset gsm8k --n_cases 100 \
--grader-type regex --threads 32
# Multiple servers (comma-separated URLs and thread counts)
python3 llama-eval.py \
--server http://server1:8033,http://server2:8033 \
--server-name server1,server2 \
--threads 16,16 \
--dataset aime2025 --n_cases 240 \
--grader-type regex
```
### Authentication
A server started with `--api-key` (or any OpenAI-compatible endpoint behind a
token) needs one here too. `--api-key` is sent as `Authorization: Bearer` on
every eval request; it falls back to `$LLAMA_API_KEY`, which keeps the secret
out of your shell history and off the process list.
```bash
# One key for every server
python3 llama-eval.py --server http://localhost:8033 --api-key sk-... ...
# From the environment
export LLAMA_API_KEY=sk-...
python3 llama-eval.py --server http://localhost:8033 ...
# One key per server, in --server order
python3 llama-eval.py \
--server http://server1:8033,http://server2:8033 \
--api-key sk-one,sk-two --threads 16,16 ...
```
The LLM grader authenticates too. It reuses the key of the server it runs on,
so `--grader-type llm` without `--grader-server` needs nothing extra; a grader
on a separate endpoint takes `--grader-api-key`.
Endpoints on an opencode host additionally get an `x-opencode-session` header,
generated fresh for every task. An agentic episode sends the same id on all of
its turns, so opencode sees one session per task instead of one per request;
other servers never see the header.
## Agentic suite
`--dataset agentic` is a different shape from the other code suites. It runs
**SACB** (Simple Agent Coding Benchmark), hosted at
[`ilintar/SACB`](https://huggingface.co/datasets/ilintar/SACB) -- 60 tasks over
three hand-written repositories, in Python and TypeScript. Rather than
answering one prompt, the model is given a small but real repository, a
deliberately narrow tool set, and a bug report, and drives its own multi-turn
conversation until it declares itself finished. It is then graded by running a
test suite it was never allowed to see.
```bash
python3 llama-eval.py \
--server http://localhost:8033 --model my-model \
--dataset agentic --threads 1 --temperature 0
```
### The tools
| tool | purpose |
|---|---|
| `list_files` | enumerate the tree |
| `read_file` | read, with 1-based line numbers and optional line-range narrowing |
| `search` | regular-expression search across the repository |
| `edit_lines` | replace an inclusive line range |
| `edit_replace` | replace an exact snippet, which must be unique |
| `write_file` | write a whole file |
| `lint` | run the language's linter and type checker |
| `finish` | declare the change complete |
Three constraints are deliberate:
**There is no shell and no way to run the tests.** The only feedback channel is
`lint`. A syntax or type error can be driven out mechanically; a logic error has
to be reasoned about. A harness that lets an agent iterate against the test
suite measures something closer to search than to understanding.
**There are two edit tools**, one line-addressed and one content-addressed.
Which one a model reaches for, and whether it keeps line numbers straight after
its own earlier edit, is itself a signal.
**The tests never touch the working tree.** They are held outside it and copied
into a throwaway copy only after the agent has stopped, so they can be neither
read nor edited. Every tool path is resolved and confinement-checked against the
repository root, so a symlink planted inside the tree cannot reach outside it.
### Scoring
A task is **resolved** when every test that was failing now passes *and* nothing
that was already passing broke. Both sets are derived by running the tests twice
when the corpus is built -- once against the defective tree and once against the
reference fix -- never hand-written, so a task whose defect no test exercises is
rejected rather than shipped.
Because many tasks contain several independent defects, the fraction of target
tests passed is reported alongside, which separates "fixed two of the three
faults" from "changed nothing". Process metrics are reported too: turns, tool
calls, rejected calls, edits, and the peak conversation size.
### Dependencies
Nothing here is installed unless you select this suite, and then only for the
language you actually run. `--agentic-lang python` provisions ruff and mypy;
`--agentic-lang typescript` provisions the TypeScript compiler and uses node's
own test runner. Neither touches the other, and no other suite touches either.
### Useful flags
| flag | meaning |
|---|---|
| `--agentic-lang` | restrict to one language, and provision only its toolchain |
| `--agentic-max-turns` | give up after this many assistant turns (default 50) |
| `--agentic-max-tokens` | cap generation per turn (default: no cap); bounds a model that falls into a repetition loop, but also cuts reasoning off mid-thought. Truncated turns are counted as `truncated_turns` |
| `--agentic-context-limit` | abandon an episode once its context exceeds this many tokens |
| `--agentic-transcripts DIR` | save each episode's full message history to `DIR/<task_id>.json`, plus a `DIR/<task_id>.html` showing every reasoning block, reply, tool call and tool result; the main report links to it |
## Code suites (HumanEval, ClassEval)
`--dataset humaneval` runs the 164 OpenAI HumanEval problems; `--dataset
classeval` runs the 100 ClassEval class-generation tasks. Unlike the other
suites these are not graded by matching text: the model's reply is parsed for a
```` ```python ```` block, that code is combined with the task's own unit tests,
and the result is **executed**.
ClassEval asks for a whole class rather than one function, so it is scored the
way upstream does - **class level** (every one of the task's tests passes, which
is what feeds pass@k) and **method level** (what fraction of all 2196 test
methods passed). The second number separates "generated nothing usable" from
"got most of the class right", and both are printed at the end of a run.
Give it more room to generate than HumanEval - a class with ten methods does not
fit in a short budget:
```bash
python3 llama-eval.py \
--server http://localhost:8033 --model my-model \
--dataset classeval --threads 8 --n_predict 4096 --temperature 0
```
```bash
# pass@1, greedy
python3 llama-eval.py \
--server http://localhost:8033 --model my-model \
--dataset humaneval --threads 8 --n_predict 1024 --temperature 0
# pass@10 -- each extra multiple of 164 is another sample per problem
python3 llama-eval.py \
--server http://localhost:8033 --model my-model \
--dataset humaneval --n_cases 1640 --temperature 0.8 --top-p 0.95
```
`--grader-type exec` is the default for code suites, so neither it nor
`--dataset-source` normally needs to be passed.
### Execution sandbox - no Docker required
Generated code is untrusted, but these suites need far less than a container to
run safely. **The package set is derived from the dataset's own imports at
startup**, not hardcoded and not taken from the upstream `requirements.txt` -
which for ClassEval lists `torch`, `openai` and `transformers` that its *tests*
never touch. So each suite installs exactly what it needs:
| suite | third-party packages |
|---|---|
| HumanEval | **none** - every problem imports only stdlib (`typing, math, random, copy, string, collections, hashlib, re`) |
| ClassEval | 13: `beautifulsoup4, gensim, lxml, netifaces, nltk, numpy, openpyxl, pandas, pillow, pypdf2, python-docx, reportlab, scipy` |
Two of ClassEval's are *declared* rather than derived, because no import
statement mentions them: `lxml` is selected by string
(`BeautifulSoup(html, 'lxml')`, ClassEval_44 - without it that task silently
drops from 23/23 to 7/23), and `scipy` is a hard gensim requirement. Note also
that `PyPDF2` must **not** be modernised to `pypdf`: ClassEval_69's test imports
`PdfFileReader`, a name pypdf removed.
The venv is cached under `~/.cache/llama-eval/` (override with
`LLAMA_EVAL_CACHE`) and rebuilt only when the package set, interpreter version
or provisioning steps change; force one with `--rebuild-sandbox`. `uv` is used
when available, which makes even the ClassEval venv build in seconds.
**Interpreter selection.** A suite can prefer a different Python than the one
running the tool, and declares it: ClassEval asks for 3.13, because gensim
publishes no wheel past cp313 and its generated C no longer compiles on 3.14
(it still uses `PyLongObject.ob_digit`). That interpreter is found automatically,
including via `uv python`; `--sandbox-python` overrides it.
3.14 is fully supported anyway. ClassEval uses exactly two gensim APIs
(`utils.decode_htmlentities`, `matutils.unitvec`), both pure Python upstream, so
on 3.14+ the suite drops gensim from the install set and provisions a small shim
instead. Both paths score **100/100**.
**Offline provisioning.** A suite that would otherwise fetch data at test time
declares a provisioning step that runs once at venv build time, while the
network is still up. ClassEval uses this to pre-seed exactly three nltk corpora
(`punkt_tab`, `averaged_perceptron_tagger_eng`, `wordnet`, ~33 MB), so the tests
themselves run with no network at all. Those are the *current* names - the
dataset asks for `punkt` and `averaged_perceptron_tagger`, which nltk has since
renamed, and seeding the legacy names too is not harmless: with a network
available nltk considers them satisfied and re-fetches the modern ones anyway.
The corpora are written *inside* the venv, not `~/nltk_data`, because `$HOME` is
a tmpfs under the strongest isolation tier.
Each candidate program runs in a throwaway directory as its own process group,
under the strongest isolation the host offers:
| tier | requires | isolates |
|---|---|---|
| `bwrap` | bubblewrap installed | read-only root, tmpfs `/tmp` and `$HOME`, **no network** |
| `unshare` | `unshare --map-current-user` | **no network** |
| `unshare-root` | any `unshare -r` | no network, but **runs as uid 0** - see below |
| `rlimit` | always available | cpu/memory/file-size/process caps only |
**Candidate code must not run as root**, which is why `unshare -r` is a
last-resort tier that prints a warning. Root bypasses file permission bits, so a
test asserting that writing to a `chmod 0444` file *fails* instead sees it
succeed - ClassEval_50 scores 14/16 under uid 0 and 16/16 under uid 1000, with
nothing in the output to suggest the sandbox caused it. Install bubblewrap.
All tiers additionally cap CPU time, address space (4 GiB), file size and core
dumps, and kill the whole process group on timeout (`--exec-timeout`, default
15 s - raise it for ClassEval, whose tests do real work). Writes to the
throwaway working directory are allowed, which is all the file-handling tasks in
ClassEval need.
Note that the weakest tier does not contain filesystem access. If you are
scoring output from a model you do not trust, check the tier printed at startup.
### Reproducibility
Several tasks draw their test inputs from the *global, unseeded* RNG, so a
verdict is not reproducible run to run - which adds noise exactly when you are
comparing two models or two quantisations. `--exec-seed` (default `0`) seeds the
RNG before each test; pass `--exec-seed -1` for the stock upstream behaviour.
This does not change what the model is asked to compute.
It matters more than it sounds. HumanEval/38, /50 and /53 are affected, and
ClassEval_58 (duplicate mine placement) fails about 30% of the time on its own
*reference* solution - measured here at 21/30 passes unseeded versus **30/30
seeded**.
### Dataset fidelity
The canonical dataset runs **unmodified on current Python** - all 164 reference
solutions pass on CPython 3.14 with no patches, so no forked or re-hosted copy
is needed. `--dataset-source` accepts a local `.jsonl` or a HuggingFace repo id
if you do want to point at a modified copy.
HumanEval does contain a handful of long-standing defects where the *prompt*
contradicts the graded test (notably HumanEval/47, /116 and /148). Those
penalise a model that reads the docstring carefully, but correcting them makes
the benchmark marginally easier and the resulting scores no longer comparable
with published HumanEval numbers, so nothing is patched by default.
ClassEval is different: the upstream data has defects that cap the achievable
score below 100% for reasons unrelated to the model, so `--dataset classeval`
defaults to a **patched fork**,
[`ilintar/ClassEval`](https://huggingface.co/datasets/ilintar/ClassEval).
Six edits over five tasks lift the reference solutions from 96/100 classes and
2182/2196 test methods to **100/100 and 2196/2196**. Three repair tests nothing
could pass:
- **ClassEval_17** was a time bomb - a hard-coded 2024 date checked against
`datetime.now()`, unpassable since January 2024.
- **ClassEval_48** asserted `get_hostname('0.0.0.0') == 'LAPTOP-2CS86KUM'`, the
dataset author's own machine name.
- **ClassEval_31** compared floats with `assertEqual`, failing on a 4-ULP numpy
difference.
The other three fix only the *reference* solutions (NumPy 2.0 removals in
ClassEval_51, a 30%-flaky mine generator in ClassEval_58) and cannot affect a
model's score. No `skeleton` was touched, so what the model is asked to write is
byte-identical to upstream. Pass `--dataset-source FudanSELab/ClassEval` for the
unpatched original.
llama.cpp/examples/llama-eval/agentic_eval.py
+918
View File
@@ -0,0 +1,918 @@
#!/usr/bin/env python3
"""Multi-turn agentic coding eval: tools over a real working tree.
The existing code suites are single-shot -- one prompt, one reply, execute,
grade. This one is not. The model is given a small but real repository and a
deliberately narrow tool set, and has to find its way to the defect before it
can fix it. That shape is what puts a run in the tens-of-thousands-of-tokens
range without any artificial padding: the context grows because the model
chose to read things.
Three deliberate constraints:
* No shell, and no way to run the tests. The only feedback channel is `lint`.
A syntax or type error can be driven out mechanically; a logic error has to
be reasoned about. Suites that let an agent iterate against the test suite
measure something closer to search than to understanding.
* Two edit tools, one line-addressed and one content-addressed. Which one a
model reaches for, and whether it keeps line numbers straight after its own
earlier edit, is itself a signal.
* Tests never touch the working tree. They are held outside it and copied in
after the agent has stopped, so they can be neither read nor edited.
Everything here is imported lazily by llama-eval.py, and every dependency it
needs is scoped to the language actually under test, so none of it is
installed for anyone running the other suites.
"""
import html
import json
import os
import re
import shutil
import tempfile
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Dict, List, Optional, Sequence, Tuple
MAX_READ_LINES = 400
MAX_TOOL_CHARS = 12000
MAX_SEARCH_HITS = 60
@dataclass
class LangSpec:
"""Everything language-specific: how to lint, how to test, what to install.
Kept as data so adding a language is a corpus concern rather than a harness
change, and so a run only ever provisions the toolchain for the language it
is actually evaluating.
"""
name: str
exts: Tuple[str, ...]
packages: Tuple[str, ...] = ()
provision: Tuple[str, ...] = ()
env: Dict[str, str] = field(default_factory=dict)
address_space_limit: Optional[int] = 4 * 1024 * 1024 * 1024
def source_files(self, root: Path) -> List[Path]:
return sorted(p for p in root.rglob("*")
if p.is_file() and p.suffix in self.exts)
PYTHON = LangSpec(
name="python",
exts=(".py",),
packages=("ruff==0.14.2", "mypy==1.18.2"),
)
TYPESCRIPT = LangSpec(
name="typescript",
exts=(".ts",),
address_space_limit=16 * 1024 * 1024 * 1024,
provision=(
"import os, subprocess, shutil, sys\n"
"npm = shutil.which('npm')\n"
"if not npm: sys.exit('npm not found on PATH; needed for the typescript suite')\n"
"dest = os.path.join(os.environ['SANDBOX_VENV'], 'node')\n"
"os.makedirs(dest, exist_ok=True)\n"
"r = subprocess.run([npm, 'install', '--silent', '--no-audit', '--no-fund',\n"
" '--prefix', dest, 'typescript@5.9.3'],\n"
" capture_output=True, text=True)\n"
"sys.exit(0 if r.returncode == 0 else (r.stderr or r.stdout)[-2000:])\n",
),
)
LANGS: Dict[str, LangSpec] = {"python": PYTHON, "typescript": TYPESCRIPT}
class PathEscape(Exception):
"""A tool was handed a path outside the repository."""
class Workspace:
"""A private, writable copy of one task's repository.
Every tool path is resolved through here. Confinement is checked on the
fully-resolved path, so a symlink planted inside the tree cannot be used to
read or write outside it.
"""
def __init__(self, root: Path):
self.root = root.resolve()
self.escape_attempts = 0
def resolve(self, rel: str) -> Path:
if rel is None or not str(rel).strip():
raise PathEscape("no path given")
raw = Path(str(rel))
base = raw if raw.is_absolute() else (self.root / raw)
candidate = base.resolve()
if candidate != self.root and self.root not in candidate.parents:
self.escape_attempts += 1
raise PathEscape(f"path {rel!r} is outside the repository")
return candidate
def rel(self, path: Path) -> str:
return str(path.relative_to(self.root))
def files(self) -> List[Path]:
return sorted(p for p in self.root.rglob("*")
if p.is_file() and "__pycache__" not in p.parts)
def snapshot(self) -> Dict[str, str]:
out = {}
for p in self.files():
try:
out[self.rel(p)] = p.read_text()
except (UnicodeDecodeError, OSError):
pass
return out
def _clip(text: str, limit: int = MAX_TOOL_CHARS) -> str:
if len(text) <= limit:
return text
return text[:limit] + f"\n... [truncated, {len(text) - limit} more characters]"
def _numbered(lines: Sequence[str], start: int) -> str:
width = len(str(start + len(lines) - 1))
return "\n".join(f"{start + i:>{width}}\t{ln}" for i, ln in enumerate(lines))
def tool_schemas() -> List[Dict[str, Any]]:
"""OpenAI-format function schemas for the whole tool set."""
def fn(name, desc, props, required=()):
return {"type": "function", "function": {
"name": name, "description": desc,
"parameters": {"type": "object", "properties": props,
"required": list(required)}}}
return [
fn("list_files", "List files in the repository.",
{"path": {"type": "string",
"description": "Directory relative to the repository root. "
"Omit for the whole tree."}}),
fn("read_file",
"Read a file. Output is prefixed with 1-based line numbers. Give "
"start_line/end_line to read only part of a large file.",
{"path": {"type": "string", "description": "File path relative to the repository root."},
"start_line": {"type": "integer", "description": "First line to read, 1-based, inclusive."},
"end_line": {"type": "integer", "description": "Last line to read, 1-based, inclusive."}},
["path"]),
fn("search",
"Search the repository with a regular expression. Returns matching "
"lines with their file and line number.",
{"pattern": {"type": "string", "description": "Python regular expression."},
"path": {"type": "string", "description": "Limit the search to this subdirectory."},
"glob": {"type": "string", "description": "Limit to files matching this glob, e.g. '*.py'."}},
["pattern"]),
fn("edit_lines",
"Replace an inclusive 1-based line range with new text. Line numbers "
"refer to the file as it is now, so re-read it after any edit that "
"changed the number of lines.",
{"path": {"type": "string", "description": "File path relative to the repository root."},
"start_line": {"type": "integer", "description": "First line to replace, 1-based, inclusive."},
"end_line": {"type": "integer", "description": "Last line to replace, 1-based, inclusive."},
"new_text": {"type": "string", "description": "Replacement text. Use an empty string to delete the range."}},
["path", "start_line", "end_line", "new_text"]),
fn("edit_replace",
"Replace an exact snippet of text. The snippet must occur exactly "
"once in the file unless replace_all is set.",
{"path": {"type": "string", "description": "File path relative to the repository root."},
"old_text": {"type": "string", "description": "Exact text to find, including indentation."},
"new_text": {"type": "string", "description": "Replacement text."},
"replace_all": {"type": "boolean", "description": "Replace every occurrence instead of requiring exactly one."}},
["path", "old_text", "new_text"]),
fn("write_file",
"Write a whole file, creating or overwriting it. Prefer the edit "
"tools for changes to existing files.",
{"path": {"type": "string", "description": "File path relative to the repository root."},
"content": {"type": "string", "description": "Full new file contents."}},
["path", "content"]),
fn("lint",
"Run the language's linter and type checker over the repository and "
"return the diagnostics. This is the only feedback available; the "
"test suite cannot be run.",
{"path": {"type": "string", "description": "Limit to this file or directory."}}),
fn("finish",
"Declare the task complete and end the session. This is the only "
"way to end it; call this once the change is made.",
{"summary": {"type": "string", "description": "Brief description of the change made."}}),
]
class ToolBox:
"""Executes tool calls against one workspace. Returns plain text results.
Failures are reported as ordinary text rather than raised, because how a
model recovers from a rejected edit is part of what is being measured.
"""
def __init__(self, ws: Workspace, linter=None):
self.ws = ws
self.linter = linter
self.calls = 0
self.errors = 0
self.edits = 0
self.finished = False
self.finish_summary: Optional[str] = None
self.per_tool: Dict[str, int] = {}
def dispatch(self, name: str, args: Dict[str, Any]) -> str:
self.calls += 1
self.per_tool[name] = self.per_tool.get(name, 0) + 1
handler = getattr(self, f"_t_{name}", None)
if handler is None:
result = f"Error: unknown tool {name!r}."
else:
try:
result = handler(args)
except PathEscape as e:
result = f"Error: {e}"
except KeyError as e:
result = f"Error: missing required argument {e}."
except Exception as e:
result = f"Error: {type(e).__name__}: {e}"
if result.startswith("Error:"):
self.errors += 1
return _clip(result)
def _t_list_files(self, a):
base = self.ws.resolve(a.get("path") or ".")
if not base.exists():
return f"Error: {a.get('path')!r} does not exist."
paths = [p for p in self.ws.files() if p == base or base in p.parents]
if not paths:
return "(no files)"
return "\n".join(f"{self.ws.rel(p)}\t{p.stat().st_size} bytes" for p in paths)
def _t_read_file(self, a):
path = self.ws.resolve(a["path"])
if not path.is_file():
return f"Error: {a['path']!r} is not a file."
lines = path.read_text().splitlines()
start = max(1, int(a.get("start_line") or 1))
end = int(a.get("end_line") or len(lines))
end = min(end, len(lines))
if start > len(lines):
return f"Error: {a['path']!r} has only {len(lines)} lines."
window = lines[start - 1:end]
note = ""
if len(window) > MAX_READ_LINES:
window = window[:MAX_READ_LINES]
note = (f"\n... [showing {MAX_READ_LINES} of {end - start + 1} requested "
f"lines; re-read with start_line/end_line for the rest]")
header = f"{a['path']} ({len(lines)} lines total)\n"
return header + _numbered(window, start) + note
def _t_search(self, a):
try:
rx = re.compile(a["pattern"])
except re.error as e:
return f"Error: bad regular expression: {e}"
base = self.ws.resolve(a.get("path") or ".")
glob = a.get("glob")
hits, scanned = [], 0
for p in self.ws.files():
if not (p == base or base in p.parents):
continue
if glob and not p.match(glob):
continue
try:
text = p.read_text()
except (UnicodeDecodeError, OSError):
continue
scanned += 1
for i, line in enumerate(text.splitlines(), 1):
if rx.search(line):
hits.append(f"{self.ws.rel(p)}:{i}:{line.strip()[:200]}")
if len(hits) >= MAX_SEARCH_HITS:
return ("\n".join(hits) +
f"\n... [stopped at {MAX_SEARCH_HITS} matches; narrow the pattern]")
if not hits:
return f"No matches for {a['pattern']!r} in {scanned} file(s)."
return "\n".join(hits)
def _t_edit_lines(self, a):
path = self.ws.resolve(a["path"])
if not path.is_file():
return f"Error: {a['path']!r} is not a file."
lines = path.read_text().splitlines()
start, end = int(a["start_line"]), int(a["end_line"])
if start < 1 or start > len(lines):
return (f"Error: start_line {start} out of range; "
f"{a['path']!r} has {len(lines)} lines.")
if end < start or end > len(lines):
return (f"Error: end_line {end} out of range; must be between "
f"{start} and {len(lines)}.")
new = str(a["new_text"]).splitlines()
path.write_text("\n".join(lines[:start - 1] + new + lines[end:]) + "\n")
self.edits += 1
delta = len(new) - (end - start + 1)
return (f"Replaced lines {start}-{end} of {a['path']} "
f"({end - start + 1} -> {len(new)} lines, {delta:+d}). "
f"Later line numbers have shifted." if delta else
f"Replaced lines {start}-{end} of {a['path']}.")
def _t_edit_replace(self, a):
path = self.ws.resolve(a["path"])
if not path.is_file():
return f"Error: {a['path']!r} is not a file."
text = path.read_text()
old, new = str(a["old_text"]), str(a["new_text"])
if not old:
return "Error: old_text must not be empty."
n = text.count(old)
if n == 0:
return (f"Error: old_text not found in {a['path']}. It must match "
f"exactly, including indentation.")
if n > 1 and not a.get("replace_all"):
return (f"Error: old_text occurs {n} times in {a['path']}. Include "
f"more surrounding context to make it unique, or set "
f"replace_all.")
path.write_text(text.replace(old, new))
self.edits += 1
return f"Replaced {n} occurrence(s) in {a['path']}."
def _t_write_file(self, a):
path = self.ws.resolve(a["path"])
path.parent.mkdir(parents=True, exist_ok=True)
existed = path.is_file()
path.write_text(str(a["content"]))
self.edits += 1
return f"{'Overwrote' if existed else 'Created'} {a['path']}."
def _t_lint(self, a):
if self.linter is None:
return "Error: no linter is configured for this task."
target = a.get("path")
if target:
self.ws.resolve(target)
return self.linter(target)
def _t_finish(self, a):
self.finished = True
self.finish_summary = a.get("summary")
return "Task marked complete."
PY_TEST_RUNNER = r'''
import json, os, random, sys, unittest
sys.path.insert(0, os.getcwd())
random.seed(0)
class Collect(unittest.TestResult):
def __init__(self):
super().__init__()
self.outcomes = {}
def addSuccess(self, t): self.outcomes[t.id()] = "pass"
def addFailure(self, t, e): self.outcomes[t.id()] = "fail"
def addError(self, t, e): self.outcomes[t.id()] = "error"
def addSkip(self, t, r): self.outcomes[t.id()] = "skip"
def addExpectedFailure(self, t, e): self.outcomes[t.id()] = "pass"
def addUnexpectedSuccess(self, t): self.outcomes[t.id()] = "fail"
def flatten(s):
for t in s:
if isinstance(t, unittest.TestSuite):
yield from flatten(t)
else:
yield t
out = {"tests": {}}
try:
suite = unittest.TestLoader().discover(os.environ.get("TEST_DIR", "tests"),
top_level_dir=".")
res = Collect()
for t in flatten(suite):
tid = t.id()
try:
t.run(res)
except Exception:
pass
res.outcomes.setdefault(tid, "error")
out["tests"] = res.outcomes
except Exception as e:
out["error"] = f"{type(e).__name__}: {e}"
print("###RESULT###")
print(json.dumps(out))
'''
def _node_roots() -> List[Path]:
"""Directories that must stay visible for node to start under bwrap."""
node = shutil.which("node")
if not node:
return []
real = Path(node).resolve()
return [Path(node).parent.parent, real.parent.parent]
class Runner:
"""Runs the linter and the hidden tests for one language, in the sandbox."""
def __init__(self, sandbox, lang: LangSpec, timeout: float = 90.0):
self.sandbox = sandbox
self.lang = lang
self.timeout = timeout
def _binds(self) -> List[Path]:
return _node_roots() if self.lang.name == "typescript" else []
def _env(self) -> Dict[str, str]:
if self.lang.name != "typescript":
return {}
node = shutil.which("node")
return {"PATH": f"{Path(node).parent}:/usr/bin:/bin"} if node else {}
def lint(self, workdir: Path, target: Optional[str] = None) -> str:
if self.lang.name == "python":
return self._lint_python(workdir, target)
return self._lint_typescript(workdir, target)
def _lint_python(self, workdir: Path, target: Optional[str]) -> str:
venv = self.sandbox.venv_dir
tgt = target or "."
chunks = []
r = self.sandbox.run_argv(
[str(venv / "bin" / "ruff"), "check", "--no-cache",
"--output-format", "concise", tgt],
workdir, timeout=self.timeout)
text = (r.stdout or r.stderr).strip()
if text and "All checks passed" not in text:
chunks.append(text)
r = self.sandbox.run_argv(
[str(venv / "bin" / "mypy"), "--no-color-output", "--no-error-summary",
"--cache-dir", "/dev/null", "--ignore-missing-imports", tgt],
workdir, timeout=self.timeout)
text = (r.stdout or r.stderr).strip()
if text and not text.startswith("Success:"):
chunks.append(text)
return "\n".join(chunks) if chunks else "No problems found."
def _lint_typescript(self, workdir: Path, target: Optional[str]) -> str:
tsc = self.sandbox.venv_dir / "node" / "node_modules" / "typescript" / "bin" / "tsc"
files = [str(p.relative_to(workdir)) for p in sorted(workdir.rglob("*.ts"))
if "node_modules" not in p.parts]
if target:
t = (workdir / target).resolve()
files = [f for f in files if (workdir / f).resolve() == t
or t in (workdir / f).resolve().parents]
if not files:
return "No TypeScript files to check."
r = self.sandbox.run_argv(
["node", str(tsc), "--noEmit", "--strict", "--target", "es2022",
"--module", "esnext", "--moduleResolution", "bundler",
"--allowImportingTsExtensions", *files],
workdir, timeout=self.timeout, env=self._env(), ro_binds=self._binds())
out = (r.stdout or "") + (r.stderr or "")
return out.strip() or "No problems found."
def test(self, workdir: Path) -> Dict[str, Any]:
if self.lang.name == "python":
return self._test_python(workdir)
return self._test_typescript(workdir)
def _test_python(self, workdir: Path) -> Dict[str, Any]:
runner = workdir / "__run_tests.py"
runner.write_text(PY_TEST_RUNNER)
r = self.sandbox.run_argv(
[str(self.sandbox.python), "-I", "-B", str(runner)],
workdir, timeout=self.timeout, env={"TEST_DIR": "tests"})
return self._parse_json_result(r)
def _parse_json_result(self, r) -> Dict[str, Any]:
if r.status == "timeout":
return {"tests": {}, "error": "timeout"}
marker = "###RESULT###"
if marker not in (r.stdout or ""):
return {"tests": {},
"error": f"no result: {(r.stderr or r.stdout or '')[-400:]}"}
try:
return json.loads(r.stdout.split(marker, 1)[1].strip())
except Exception as e:
return {"tests": {}, "error": f"unparseable result: {e}"}
def _test_typescript(self, workdir: Path) -> Dict[str, Any]:
tests = sorted(p for p in (workdir / "tests").rglob("*.ts")) \
if (workdir / "tests").is_dir() else []
if not tests:
return {"tests": {}, "error": "no test files"}
r = self.sandbox.run_argv(
["node", "--test", "--experimental-strip-types",
"--test-reporter=tap",
*[str(p.relative_to(workdir)) for p in tests]],
workdir, timeout=self.timeout, env=self._env(), ro_binds=self._binds())
return self._parse_tap(r)
@staticmethod
def _parse_tap(r) -> Dict[str, Any]:
if r.status == "timeout":
return {"tests": {}, "error": "timeout"}
text = (r.stdout or "") + "\n" + (r.stderr or "")
outcomes: Dict[str, str] = {}
for line in text.splitlines():
m = re.match(r"\s*(not )?ok\s+\d+\s*-\s*(.+?)\s*$", line)
if not m:
continue
name = m.group(2).strip()
if name.endswith(" # SKIP") or " # SKIP" in name:
outcomes[name.split(" # ")[0]] = "skip"
continue
verdict = "fail" if m.group(1) else "pass"
if outcomes.get(name) != "fail":
outcomes[name] = verdict
if not outcomes:
return {"tests": {},
"error": f"no TAP output: {text.strip()[-400:]}"}
return {"tests": outcomes}
SYSTEM_PROMPT = """You are an expert {lang} engineer working in an existing repository.
Resolve the user's request by editing files with the tools provided.
How this session works:
- You are working autonomously. There is no human watching; nobody will answer
questions, approve a plan, or read progress reports.
- Every reply you send must contain at least one tool call until the task is
done. A reply that contains only text does nothing: no work happens and the
turn is wasted. Do not stop to describe what you are about to do; do it.
- The session ends only when you call `finish`. Call it once the change is
made, and not before. Only the files as you leave them are evaluated.
Important constraints:
- You cannot run the code, the tests, or any shell command. The only feedback
available is the `lint` tool, which reports syntax and type problems.
- The repository already has a test suite that will be run against your work
after you finish, but you cannot see it or run it. Do not create test files.
- Read enough of the code to understand it before editing. A change that only
makes the symptom go away will usually fail the tests.
- Make the smallest correct change. Do not reformat or refactor unrelated code.
Keep calling tools until the change is complete, then call `finish`."""
@dataclass
class AgenticTask:
task_id: str
lang: str
category: str
difficulty: int
instruction: str
files: Dict[str, str]
tests: Dict[str, str]
gold: Dict[str, str] = field(default_factory=dict)
fail_to_pass: List[str] = field(default_factory=list)
pass_to_pass: List[str] = field(default_factory=list)
@classmethod
def from_record(cls, rec: Dict[str, Any]) -> "AgenticTask":
def as_map(v):
return v if isinstance(v, dict) else json.loads(v or "{}")
def as_list(v):
return v if isinstance(v, list) else json.loads(v or "[]")
return cls(
task_id=rec["task_id"], lang=rec["lang"], category=rec["category"],
difficulty=int(rec.get("difficulty", 3)),
instruction=rec["instruction"],
files=as_map(rec["files"]), tests=as_map(rec["tests"]),
gold=as_map(rec.get("gold")),
fail_to_pass=as_list(rec.get("fail_to_pass")),
pass_to_pass=as_list(rec.get("pass_to_pass")),
)
@property
def spec(self) -> LangSpec:
return LANGS[self.lang]
def materialise(files: Dict[str, str], dest: Path) -> Path:
"""Write a file map out as a real directory tree."""
dest.mkdir(parents=True, exist_ok=True)
for rel, content in files.items():
p = dest / rel
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(content)
return dest
def repo_overview(files: Dict[str, str]) -> str:
"""A compact tree, so a run is not lost merely to not finding the files."""
lines = []
for rel in sorted(files):
n = files[rel].count("\n") + 1
lines.append(f" {rel} ({n} lines)")
return "\n".join(lines)
def user_prompt(task: AgenticTask) -> str:
return (f"Repository files:\n{repo_overview(task.files)}\n\n"
f"Task:\n{task.instruction}")
@dataclass
class EpisodeResult:
stop_reason: str = "unknown"
turns: int = 0
tool_calls: int = 0
tool_errors: int = 0
edits: int = 0
escape_attempts: int = 0
finished: bool = False
bad_tool_json: int = 0
nudges: int = 0
prompt_tokens: int = 0
completion_tokens: int = 0
peak_context: int = 0
truncated_turns: int = 0
per_tool: Dict[str, int] = field(default_factory=dict)
error: Optional[str] = None
NUDGE = ("Your last reply contained no tool call, so nothing happened. There is "
"no human in this session to respond to text. If your change is "
"complete, call `finish`. Otherwise continue working by calling the "
"tools; do not reply with text alone.")
def run_episode(task: AgenticTask, toolbox: ToolBox, chat,
max_turns: int = 50,
context_limit: Optional[int] = None,
max_nudges: int = 3) -> Tuple[EpisodeResult, List[Dict]]:
"""Drive one multi-turn tool-calling episode.
`chat(messages, tools) -> response dict` is supplied by the caller so this
module stays free of any HTTP concerns.
"""
tools = tool_schemas()
messages: List[Dict[str, Any]] = [
{"role": "system", "content": SYSTEM_PROMPT.format(lang=task.spec.name)},
{"role": "user", "content": user_prompt(task)},
]
res = EpisodeResult()
for turn in range(max_turns):
res.turns = turn + 1
try:
reply = chat(messages, tools)
except Exception as e:
res.stop_reason, res.error = "request_failed", f"{type(e).__name__}: {e}"
break
usage = reply.get("usage") or {}
pt = int(usage.get("prompt_tokens") or 0)
ct = int(usage.get("completion_tokens") or 0)
res.prompt_tokens += pt
res.completion_tokens += ct
res.peak_context = max(res.peak_context, pt + ct)
choices = reply.get("choices") or []
if not choices:
res.stop_reason, res.error = "no_choices", json.dumps(reply)[:300]
break
msg = choices[0].get("message") or {}
if choices[0].get("finish_reason") == "length":
res.truncated_turns += 1
calls = msg.get("tool_calls") or []
reasoning = msg.get("reasoning_content")
messages.append({
"role": "assistant",
"content": msg.get("content") or "",
**({"reasoning_content": reasoning} if reasoning else {}),
**({"tool_calls": calls} if calls else {}),
})
if not calls:
if toolbox.finished:
res.stop_reason = "finished"
break
if res.nudges >= max_nudges:
res.stop_reason = "no_tool_calls"
break
res.nudges += 1
messages.append({"role": "user", "content": NUDGE})
continue
for call in calls:
fn = call.get("function") or {}
name = fn.get("name") or ""
raw = fn.get("arguments")
if isinstance(raw, dict):
args = raw
else:
try:
args = json.loads(raw or "{}")
except (json.JSONDecodeError, TypeError):
res.bad_tool_json += 1
args = None
output = (toolbox.dispatch(name, args) if args is not None else
f"Error: arguments for {name!r} were not valid JSON.")
messages.append({"role": "tool",
"tool_call_id": call.get("id") or f"call_{res.tool_calls}",
"content": output})
if toolbox.finished:
res.stop_reason = "finished"
break
if context_limit and res.peak_context > context_limit:
res.stop_reason = "context_limit"
break
else:
res.stop_reason = "max_turns"
res.tool_calls = toolbox.calls
res.tool_errors = toolbox.errors
res.edits = toolbox.edits
res.escape_attempts = toolbox.ws.escape_attempts
res.finished = toolbox.finished
res.per_tool = dict(toolbox.per_tool)
return res, messages
def run_tests_against(files: Dict[str, str], task: AgenticTask, runner: Runner,
workroot: Path, label: str) -> Dict[str, str]:
"""Materialise a tree, drop the hidden tests in, and run them.
The tests are only ever written into this throwaway copy, never into the
tree the agent worked in, so there is no window in which they could have
been read or edited.
"""
dest = workroot / f"grade-{label}"
shutil.rmtree(dest, ignore_errors=True)
materialise(files, dest)
materialise(task.tests, dest)
return runner.test(dest).get("tests", {})
def score(task: AgenticTask, outcomes: Dict[str, str]) -> Dict[str, Any]:
"""Resolved means every target test passes and nothing previously passing broke."""
def passing(tid):
return outcomes.get(tid) == "pass"
f2p_ok = [t for t in task.fail_to_pass if passing(t)]
p2p_ok = [t for t in task.pass_to_pass if passing(t)]
regressions = [t for t in task.pass_to_pass if not passing(t)]
resolved = (len(f2p_ok) == len(task.fail_to_pass)
and not regressions and bool(task.fail_to_pass))
return {
"resolved": resolved,
"f2p_passed": len(f2p_ok), "f2p_total": len(task.fail_to_pass),
"p2p_passed": len(p2p_ok), "p2p_total": len(task.pass_to_pass),
"regressions": regressions[:10],
"n_regressions": len(regressions),
"partial": (len(f2p_ok) / len(task.fail_to_pass)) if task.fail_to_pass else 0.0,
}
def make_sandbox(lang_name: str, sandbox_cls, cache_root=None, base_python=None):
"""A sandbox carrying only the toolchain for `lang_name`.
Keyed per language rather than per suite: running the Python tasks must
never provision node, and vice versa.
"""
lang = LANGS[lang_name]
return sandbox_cls(
name=f"agentic-{lang_name}",
packages=list(lang.packages),
provision=list(lang.provision),
env=dict(lang.env),
address_space_limit=lang.address_space_limit,
cache_root=cache_root,
base_python=base_python,
)
def run_task(task: AgenticTask, chat, sandbox, workroot: Optional[Path] = None,
max_turns: int = 50, context_limit: Optional[int] = None,
test_timeout: float = 90.0) -> Dict[str, Any]:
"""One full attempt: fresh workspace, agent episode, then hidden tests."""
root = Path(tempfile.mkdtemp(prefix=f"agentic-{task.task_id}-",
dir=str(workroot) if workroot else None))
try:
ws_root = root / "repo"
materialise(task.files, ws_root)
ws = Workspace(ws_root)
runner = Runner(sandbox, task.spec, timeout=test_timeout)
toolbox = ToolBox(ws, linter=lambda t: runner.lint(ws_root, t))
episode, messages = run_episode(task, toolbox, chat,
max_turns=max_turns,
context_limit=context_limit)
final = ws.snapshot()
changed = sorted(k for k in final
if final[k] != task.files.get(k)) + \
sorted(k for k in task.files if k not in final)
outcomes = run_tests_against(final, task, runner, root, "final")
result = score(task, outcomes)
result.update({
"task_id": task.task_id, "lang": task.lang,
"category": task.category, "difficulty": task.difficulty,
"changed_files": changed,
"episode": {k: v for k, v in vars(episode).items()},
"messages": messages,
})
return result
finally:
shutil.rmtree(root, ignore_errors=True)
TRANSCRIPT_CSS = """
body { font-family: system-ui, sans-serif; margin: 0; padding: 16px; background: #fff; color: #222; max-width: 1100px; }
h1 { font-size: 16px; margin: 0 0 4px; }
.meta { font-size: 13px; color: #555; font-family: 'SF Mono', 'Menlo', 'Consolas', monospace; margin-bottom: 12px; white-space: pre-wrap; }
.msg { border: 1px solid #e1e4e8; border-left-width: 4px; margin: 8px 0; padding: 6px 10px; background: #fafbfc; }
.msg .role { font-size: 12px; font-weight: 600; color: #555; margin-bottom: 4px; }
.system { border-left-color: #888; }
.user { border-left-color: #0969da; }
.nudge { border-left-color: #9a6700; }
.assistant { border-left-color: #1a7f37; background: #fff; }
.tool { border-left-color: #8250df; }
.tool.err { border-left-color: #cf222e; }
pre { font-family: 'SF Mono', 'Menlo', 'Consolas', monospace; font-size: 12px; background: #fff; border: 1px solid #e1e4e8; padding: 6px 8px; margin: 4px 0; white-space: pre-wrap; word-wrap: break-word; }
.reasoning pre { color: #57606a; background: #f6f8fa; }
.call { font-size: 12px; font-weight: 600; color: #8250df; margin-top: 6px; }
details > summary { cursor: pointer; font-size: 12px; color: #555; }
.empty { font-size: 12px; color: #888; font-style: italic; }
"""
def _pre(text: str) -> str:
return f"<pre>{html.escape(text)}</pre>"
def _render_call(call: Dict[str, Any]) -> str:
fn = call.get("function") or {}
raw = fn.get("arguments")
try:
args = raw if isinstance(raw, dict) else json.loads(raw or "{}")
shown = json.dumps(args, indent=2, ensure_ascii=False)
except (json.JSONDecodeError, TypeError):
shown = f"(invalid JSON)\n{raw}"
return (f'<div class="call">&rarr; {html.escape(fn.get("name") or "?")} '
f'<span style="font-weight:400;color:#888">{html.escape(call.get("id") or "")}</span></div>'
+ _pre(shown))
def render_transcript_html(messages: List[Dict[str, Any]], title: str,
meta: str = "") -> str:
"""One page per episode: every turn, with reasoning, tool calls and results."""
names = {}
for m in messages:
for c in m.get("tool_calls") or []:
names[c.get("id")] = (c.get("function") or {}).get("name") or "?"
parts = []
turn = 0
for m in messages:
role = m.get("role") or "?"
content = m.get("content") or ""
if role == "assistant":
turn += 1
body = ""
if m.get("reasoning_content"):
body += ('<details class="reasoning" open><summary>reasoning '
f'({len(m["reasoning_content"])} chars)</summary>'
+ _pre(m["reasoning_content"]) + "</details>")
body += _pre(content) if content.strip() else ""
body += "".join(_render_call(c) for c in m.get("tool_calls") or [])
if not body:
body = '<div class="empty">(empty reply)</div>'
parts.append(f'<div class="msg assistant"><div class="role">assistant '
f'&middot; turn {turn}</div>{body}</div>')
elif role == "tool":
name = names.get(m.get("tool_call_id"), "?")
err = " err" if content.startswith("Error") else ""
parts.append(f'<div class="msg tool{err}"><div class="role">tool result '
f'&middot; {html.escape(name)}</div>{_pre(content)}</div>')
elif role == "system":
parts.append('<div class="msg system"><details><summary>system prompt'
f'</summary>{_pre(content)}</details></div>')
else:
cls = "nudge" if content == NUDGE else role
label = "user (nudge)" if content == NUDGE else role
parts.append(f'<div class="msg {html.escape(cls)}"><div class="role">'
f'{html.escape(label)}</div>{_pre(content)}</div>')
return (f'<!DOCTYPE html>\n<html>\n<head>\n<meta charset="UTF-8">\n'
f'<title>{html.escape(title)}</title>\n<style>{TRANSCRIPT_CSS}</style>\n'
f'</head>\n<body>\n<h1>{html.escape(title)}</h1>\n'
f'<div class="meta">{html.escape(meta)}</div>\n'
+ "\n".join(parts) + "\n</body>\n</html>\n")
def write_transcript(messages: List[Dict[str, Any]], dest: Path, title: str,
meta: str = "") -> Path:
"""Write `<dest>.json` (the raw message history) and `<dest>.html`."""
dest.parent.mkdir(parents=True, exist_ok=True)
json_path = dest.with_name(dest.name + ".json")
with open(json_path, "w") as f:
json.dump(messages, f, indent=2, ensure_ascii=False)
with open(dest.with_name(dest.name + ".html"), "w") as f:
f.write(render_transcript_html(messages, title, meta))
return json_path
llama.cpp/examples/llama-eval/eval_sandbox.py
+488
View File
@@ -0,0 +1,488 @@
#!/usr/bin/env python3
"""Lightweight sandboxed execution for code-generation evals (HumanEval, ClassEval).
Replaces the usual Docker-per-suite setup with a cached venv plus whatever
process isolation the host can give us without root:
tier 1 bubblewrap read-only root, tmpfs /tmp and $HOME, no network
tier 2 unshare -rn no network (home still visible)
tier 3 rlimits only last resort, still capped on cpu/memory/procs/files
Every tier also applies rlimits and runs the candidate program in a throwaway
working directory as its own process group, so a runaway solution is killed
along with anything it spawned.
"""
import json
import os
import re
import resource
import shutil
import signal
import subprocess
import sys
import tempfile
import time
import venv
from dataclasses import dataclass
from pathlib import Path
from typing import Dict, List, Optional, Sequence
DEFAULT_CACHE_ROOT = Path(
os.environ.get("LLAMA_EVAL_CACHE", Path.home() / ".cache" / "llama-eval")
)
LIMIT_AS_BYTES = 4 * 1024 * 1024 * 1024
LIMIT_CPU_SECONDS_SLACK = 5
LIMIT_NPROC = 256
LIMIT_FSIZE_BYTES = 64 * 1024 * 1024
@dataclass
class ExecResult:
passed: bool
status: str
stdout: str = ""
stderr: str = ""
returncode: Optional[int] = None
duration: float = 0.0
def summary(self, max_chars: int = 400) -> str:
"""Short human-readable reason, for grader logs."""
if self.passed:
return "ok"
if self.status == "timeout":
return "timeout"
tail = (self.stderr or self.stdout).strip()
if not tail:
return f"{self.status} (rc={self.returncode})"
lines = [ln for ln in tail.splitlines() if ln.strip()]
reason = lines[-1] if lines else tail
return reason[:max_chars]
def resolve_python(spec: Optional[str]) -> Optional[str]:
"""Turn '3.13' or a path into a usable interpreter, or None if unavailable.
Suites can need an older interpreter than the one running this tool, because
a dependency has no wheel (or no working source build) for the newest
CPython. Accepts an explicit path, a bare version, or an executable name.
"""
if not spec:
return None
candidate = Path(spec).expanduser()
if candidate.exists() and candidate.is_file():
return str(candidate)
uv = shutil.which("uv")
if uv and re.fullmatch(r"\d+\.\d+(\.\d+)?", spec):
try:
r = subprocess.run([uv, "python", "find", spec],
capture_output=True, text=True, timeout=120)
path = r.stdout.strip()
if r.returncode == 0 and path and Path(path).exists():
return path
except Exception:
pass
return shutil.which(spec) or shutil.which(f"python{spec}")
def _detect_isolation(probe_python: Optional[str] = None,
ro_binds: Sequence[Path] = ()) -> str:
"""Pick the strongest isolation tier this host supports, once.
The probe must run the *same* interpreter the suite will, with the same
binds. Probing a bare sys.executable silently downgrades the tier whenever
this tool is itself run from a venv under $HOME -- which is the usual case,
since it needs `datasets`. bwrap tmpfs-mounts the home directory, the
interpreter disappears, the probe fails, and the sandbox falls back to a
tier that does not hide $HOME at all. Nothing about the results looks wrong
when that happens, so it has to be got right here.
"""
py = probe_python or sys.executable
probe = "import sys; sys.exit(0)"
if shutil.which("bwrap"):
try:
r = subprocess.run(
_bwrap_argv(Path.cwd(), [py, "-I", "-c", probe], ro_binds=ro_binds),
capture_output=True, timeout=20,
)
if r.returncode == 0:
return "bwrap"
except Exception:
pass
if shutil.which("unshare"):
for tier, argv in (
("unshare", ["unshare", "--user", "--map-current-user", "--net"]),
("unshare-root", ["unshare", "-rn"]),
):
try:
r = subprocess.run(
[*argv, py, "-I", "-c", probe],
capture_output=True, timeout=20,
)
if r.returncode == 0:
return tier
except Exception:
continue
return "rlimit"
UNSHARE_ARGV = {
"unshare": ["unshare", "--user", "--map-current-user", "--net"],
"unshare-root": ["unshare", "-rn"],
}
def _under(path: Path, parent: str) -> bool:
return bool(parent) and (str(path) == parent
or str(path).startswith(parent.rstrip("/") + os.sep))
def _bwrap_argv(workdir: Path, inner: Sequence[str],
ro_binds: Sequence[Path] = ()) -> List[str]:
home = os.path.expanduser("~")
argv = [
"bwrap",
"--ro-bind", "/", "/",
"--dev", "/dev",
"--proc", "/proc",
"--tmpfs", "/tmp",
]
if home and home != "/":
argv += ["--tmpfs", home]
for path in ro_binds:
if _under(path, home):
argv += ["--ro-bind", str(path), str(path)]
argv += [
"--bind", str(workdir), str(workdir),
"--chdir", str(workdir),
"--unshare-all",
"--die-with-parent",
"--new-session",
"--",
]
return argv + list(inner)
def _nproc_ceiling() -> int:
"""A process cap that bounds a fork bomb without tripping on existing load.
RLIMIT_NPROC is enforced per real UID across the whole system, not per
process tree, so it has to sit above whatever the user is already running.
"""
try:
current = len(os.listdir("/proc"))
except OSError:
current = 512
return max(current * 2, 1024)
def _apply_rlimits(timeout: float, with_nproc: bool,
as_bytes: Optional[int] = LIMIT_AS_BYTES):
def _preexec():
os.setsid()
cpu = int(timeout) + LIMIT_CPU_SECONDS_SLACK
limits = [
(resource.RLIMIT_CPU, (cpu, cpu)),
(resource.RLIMIT_FSIZE, (LIMIT_FSIZE_BYTES, LIMIT_FSIZE_BYTES)),
(resource.RLIMIT_CORE, (0, 0)),
]
if as_bytes:
limits.append((resource.RLIMIT_AS, (as_bytes, as_bytes)))
if with_nproc:
ceiling = _nproc_ceiling()
limits.append((resource.RLIMIT_NPROC, (ceiling, ceiling)))
for res, lim in limits:
try:
resource.setrlimit(res, lim)
except (ValueError, OSError):
pass
return _preexec
class Sandbox:
"""A cached venv plus an isolated runner for untrusted candidate programs."""
def __init__(
self,
name: str,
packages: Optional[Sequence[str]] = None,
cache_root: Optional[Path] = None,
base_python: Optional[str] = None,
isolation: Optional[str] = None,
provision: Optional[Sequence[str]] = None,
env: Optional[Dict[str, str]] = None,
address_space_limit: Optional[int] = LIMIT_AS_BYTES,
):
self.name = name
self.packages = sorted(set(packages or []))
self.provision = list(provision or [])
self.env = dict(env or {})
# None drops the RLIMIT_AS cap entirely -- see _apply_rlimits
self.address_space_limit = address_space_limit
self.cache_root = Path(cache_root or DEFAULT_CACHE_ROOT)
self.base_python = base_python or sys.executable
self.venv_dir = self.cache_root / f"{name}-venv"
self._python: Optional[Path] = None
self._isolation = isolation
self._base_prefix: List[Path] = []
@property
def python(self) -> Path:
if self._python is None:
raise RuntimeError("Sandbox.ensure() must be called first")
return self._python
@property
def isolation(self) -> str:
if self._isolation is None:
if self._python:
py = str(self._python)
roots = [self.venv_dir, *(self._base_prefix or [])]
else:
py = sys.executable
roots = self._detect_base_prefix(Path(py))
self._isolation = _detect_isolation(py, roots)
return self._isolation
def _stamp_path(self) -> Path:
return self.venv_dir / ".llama-eval-stamp.json"
def _stamp_matches(self) -> bool:
try:
stamp = json.loads(self._stamp_path().read_text())
except Exception:
return False
return (
stamp.get("packages") == self.packages
and stamp.get("py") == self._base_version()
and stamp.get("provision") == self.provision
)
def _resolved_env(self) -> Dict[str, str]:
return {k: v.replace("{venv}", str(self.venv_dir)) for k, v in self.env.items()}
def _base_version(self) -> str:
r = subprocess.run(
[self.base_python, "-c",
"import sys;print('.'.join(map(str,sys.version_info[:3])))"],
capture_output=True, text=True,
)
return r.stdout.strip()
def ensure(self, force: bool = False, quiet: bool = False) -> Path:
"""Create/refresh the venv. Idempotent and cheap when already valid."""
py = self.venv_dir / "bin" / "python"
if force and self.venv_dir.exists():
shutil.rmtree(self.venv_dir)
if py.exists() and self._stamp_matches():
self._python = py
self._base_prefix = self._detect_base_prefix(py)
return py
if not quiet:
pkgs = ", ".join(self.packages) if self.packages else "stdlib only"
print(f"[sandbox] provisioning venv {self.venv_dir} ({pkgs})")
self.cache_root.mkdir(parents=True, exist_ok=True)
if self.venv_dir.exists():
shutil.rmtree(self.venv_dir)
uv = shutil.which("uv")
if uv:
subprocess.run(
[uv, "venv", "--python", self.base_python, str(self.venv_dir)],
check=True, capture_output=True,
)
elif Path(self.base_python).resolve() == Path(sys.executable).resolve():
venv.EnvBuilder(with_pip=bool(self.packages), clear=True).create(self.venv_dir)
else:
subprocess.run(
[self.base_python, "-m", "venv", str(self.venv_dir)],
check=True, capture_output=True,
)
if self.packages:
if uv:
cmd = [uv, "pip", "install", "--python", str(py), *self.packages]
else:
cmd = [str(py), "-m", "pip", "install", "--quiet",
"--disable-pip-version-check", *self.packages]
r = subprocess.run(cmd, capture_output=True, text=True)
if r.returncode != 0:
raise RuntimeError(
f"failed to install {len(self.packages)} package(s) into "
f"{self.venv_dir}:\n{(r.stderr or r.stdout)[-3000:]}"
)
for i, snippet in enumerate(self.provision, 1):
if not quiet:
print(f"[sandbox] provisioning step {i}/{len(self.provision)}")
prov_env = dict(os.environ)
prov_env["SANDBOX_VENV"] = str(self.venv_dir)
prov_env.update(self._resolved_env())
r = subprocess.run([str(py), "-c", snippet], env=prov_env,
capture_output=True, text=True, timeout=1800)
if r.returncode != 0:
raise RuntimeError(
f"provisioning step {i} failed:\n{(r.stderr or r.stdout)[-3000:]}"
)
self._stamp_path().write_text(json.dumps({
"packages": self.packages,
"py": self._base_version(),
"provision": self.provision,
}))
self._python = py
self._base_prefix = self._detect_base_prefix(py)
if not quiet:
print(f"[sandbox] ready ({self.isolation} isolation)")
if self.isolation == "unshare-root":
print("Warning: this host's unshare cannot keep the caller's uid, so "
"candidate code runs as root inside the namespace. Tests that "
"assert a permission denial will not see one. Install "
"bubblewrap (bwrap) for correct results.")
return py
@staticmethod
def _detect_base_prefix(py: Path) -> List[Path]:
"""Every directory that must stay visible for the venv to start.
A venv is only symlinks into its base interpreter, so the whole install
tree has to survive the tmpfs that hides $HOME -- and binding just the
resolved tree is not enough. Version-managed interpreters (uv, pyenv)
are commonly reached through an unversioned alias symlink
(cpython-3.13-... -> cpython-3.13.14-...), and startup resolves through
the alias path, so both spellings need to be mounted. Walk the chain
and keep every install root it passes through.
"""
roots: List[Path] = []
def add(path: Path):
if path.exists() and path not in roots:
roots.append(path)
cur, hops = py, 0
while hops < 16:
hops += 1
add(cur.parent.parent)
if cur.is_symlink():
target = Path(os.readlink(cur))
cur = target if target.is_absolute() else (cur.parent / target)
else:
break
try:
r = subprocess.run([str(py), "-c", "import sys; print(sys.base_prefix)"],
capture_output=True, text=True, timeout=60)
if r.returncode == 0 and r.stdout.strip():
add(Path(r.stdout.strip()))
except Exception:
pass
return [p for p in roots if str(p) not in ("/", "/usr", "/usr/local")]
def _isolate(self, argv: Sequence[str], workdir: Path,
ro_binds: Sequence[Path] = ()) -> List[str]:
"""Wrap a command in the strongest isolation tier this host supports."""
if self.isolation == "bwrap":
return _bwrap_argv(
workdir, argv,
ro_binds=[self.venv_dir, *(self._base_prefix or []), *ro_binds],
)
if self.isolation in UNSHARE_ARGV:
return [*UNSHARE_ARGV[self.isolation], *argv]
return list(argv)
def _base_env(self, workdir: Path) -> Dict[str, str]:
env = {
"PATH": "/usr/bin:/bin",
"HOME": str(workdir),
"TMPDIR": str(workdir),
"LC_ALL": "C.UTF-8",
"PYTHONHASHSEED": "0",
"PYTHONDONTWRITEBYTECODE": "1",
"OPENBLAS_NUM_THREADS": "1",
"OMP_NUM_THREADS": "1",
}
env.update(self._resolved_env())
return env
def run_argv(self, argv: Sequence[str], workdir: Path,
timeout: float = 15.0, env: Optional[Dict[str, str]] = None,
ro_binds: Sequence[Path] = ()) -> ExecResult:
"""Run an arbitrary command in `workdir`, which stays writable.
Unlike run(), the caller owns the directory and it survives the call.
That is what a multi-step agent needs: its edits have to persist across
many tool invocations before anything is graded. Taking an argv rather
than a source string is what lets a suite drive a linter or a test
runner for a language other than Python.
"""
run_env = self._base_env(workdir)
if env:
run_env.update(env)
started = time.monotonic()
try:
proc = subprocess.Popen(
self._isolate(argv, workdir, ro_binds),
cwd=str(workdir),
stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
errors="replace",
env=run_env,
preexec_fn=_apply_rlimits(timeout,
with_nproc=self.isolation == "rlimit",
as_bytes=self.address_space_limit),
)
except Exception as e:
return ExecResult(False, "error", "", f"{type(e).__name__}: {e}")
try:
out, err = proc.communicate(timeout=timeout)
except subprocess.TimeoutExpired:
_kill_tree(proc)
out, err = proc.communicate()
return ExecResult(False, "timeout", out or "", err or "",
None, time.monotonic() - started)
duration = time.monotonic() - started
if proc.returncode == 0:
return ExecResult(True, "ok", out, err, 0, duration)
return ExecResult(False, "failed", out, err, proc.returncode, duration)
def run(self, code: str, timeout: float = 15.0,
env: Optional[Dict[str, str]] = None) -> ExecResult:
"""Run `code` to completion in the sandbox. Exit 0 == passed."""
workdir = Path(tempfile.mkdtemp(prefix=f"{self.name}-", dir="/tmp"))
try:
prog = workdir / "candidate.py"
prog.write_text(code)
return self.run_argv([str(self.python), "-I", "-B", str(prog)],
workdir, timeout=timeout, env=env)
except Exception as e:
return ExecResult(False, "error", "", f"{type(e).__name__}: {e}")
finally:
shutil.rmtree(workdir, ignore_errors=True)
def _kill_tree(proc: subprocess.Popen):
"""Kill the candidate and everything it spawned."""
for sig in (signal.SIGKILL,):
try:
os.killpg(os.getpgid(proc.pid), sig)
except (ProcessLookupError, PermissionError):
try:
proc.kill()
except ProcessLookupError:
pass
llama.cpp/examples/llama-eval/llama-eval.py
Executable
+2798
View File
File diff suppressed because it is too large. Load diff
llama.cpp/examples/llama-eval/llama-server-simulator.py
Executable
+452
View File
@@ -0,0 +1,452 @@
#!/usr/bin/env python3
import argparse
import json
import random
import re
import time
import sys
import os
import threading
from http.server import HTTPServer, BaseHTTPRequestHandler
from typing import Dict, List, Optional
from dataclasses import dataclass
from pathlib import Path
# Set cache directory for HuggingFace datasets
cache_dir = Path.home() / ".cache" / "huggingface" / "datasets"
cache_dir.mkdir(parents=True, exist_ok=True)
os.environ["HF_DATASETS_CACHE"] = str(cache_dir)
def dice(s1: str, s2: str) -> float:
"""Calculate Dice coefficient between two strings based on bigram overlap."""
if not s1 and not s2:
return 1.0
def _bigrams(s: str):
return [s[i : i + 2] for i in range(len(s) - 1)]
bigrams1 = _bigrams(s1)
bigrams2 = _bigrams(s2)
if not bigrams1 and not bigrams2:
return 1.0
from collections import Counter
freq1 = Counter(bigrams1)
freq2 = Counter(bigrams2)
intersection = sum(min(freq1[bg], freq2[bg]) for bg in freq1)
dice_coeff = 2 * intersection / (len(bigrams1) + len(bigrams2))
return dice_coeff
def debug_log(message: str):
"""Log debug messages to both stdout and a file"""
print(message, file=sys.stderr)
with open("/tmp/simulator-debug.log", "a") as f:
f.write(message + "\n")
simulator: Optional["Simulator"] = None
@dataclass
class EvalState:
id: str
tasks: List[str]
task_states: Dict[str, Dict]
sampling_config: Dict
def normalize_number(s: str) -> Optional[int]:
match = re.match(r"\d+", s) # match digits from the start
if not match:
return None
return int(match.group(0))
class AimeDataset:
def __init__(self, split: str = "train", dataset_type: str = "aime"):
self.split = split
self.dataset_type = dataset_type
self.questions: List[Dict] = []
self._load_dataset()
def _get_question_text(self, question: Dict) -> str:
"""Get question text, handling different dataset field names."""
return question.get("problem", question.get("question", ""))
def _load_dataset(self):
import datasets
if self.dataset_type == "aime":
print(f"Loading AIME dataset (split: {self.split})...")
cache_path = Path.home() / ".cache" / "huggingface" / "datasets" / "AI-MO___aimo-validation-aime" / "default" / "0.0.0"
if cache_path.exists():
print(f"Using cached dataset from {cache_path}")
ds = datasets.load_dataset("AI-MO/aimo-validation-aime", split=self.split, cache_dir=str(cache_path))
else:
ds = datasets.load_dataset("AI-MO/aimo-validation-aime", split=self.split)
elif self.dataset_type == "aime2025":
print(f"Loading AIME2025 dataset...")
ds_list = []
for config_name in ["AIME2025-I", "AIME2025-II"]:
cache_path = Path.home() / ".cache" / "huggingface" / "datasets" / "opencompass___AIME2025" / "default" / "0.0.0"
if cache_path.exists():
print(f"Using cached dataset from {cache_path}")
ds = datasets.load_dataset("opencompass/AIME2025", config_name, split="test", cache_dir=str(cache_path))
else:
ds = datasets.load_dataset("opencompass/AIME2025", config_name, split="test")
ds_list.extend(ds)
ds = ds_list
else:
raise ValueError(f"Unknown dataset type: {self.dataset_type}")
self.questions = list(ds)
print(f"{self.dataset_type} dataset loaded: {len(self.questions)} questions")
def find_question(self, request_text: str) -> Optional[Dict]:
# Strip common template prefixes to get the actual question text
# Templates include things like "Solve the following math problem step by step..."
# The actual question usually follows a blank line or after the template instruction
cleaned = request_text
# Split on double newline and take the part that looks like the problem
parts = cleaned.split('\n\n')
if len(parts) > 1:
# Find the part that's longest (likely the actual problem text)
problem_parts = [p for p in parts if len(p.strip()) > 100]
if problem_parts:
cleaned = max(problem_parts, key=lambda x: len(x))
best_match = None
best_distance = -1
best_index = -1
for i, question in enumerate(self.questions):
question_text = self._get_question_text(question)
request_lower = cleaned.lower()
question_lower = question_text.lower()
# Check if question text is contained in the cleaned request
if question_lower in request_lower or request_lower in question_lower:
debug_log(f"DEBUG: Found substring match at index {i}")
return question
# Exact match
if question_lower == request_lower:
debug_log(f"DEBUG: Found exact match at index {i}")
return question
# Remove LaTeX formatting for more flexible matching
question_no_latex = re.sub(r'\$[^$]+\$', '', question_text)
if question_no_latex.lower() == request_lower:
debug_log(f"DEBUG: Found match (no LaTeX) at index {i}")
return question
# Calculate Dice coefficient for partial matches
# Only consider if request is at least 50% of question length
if len(request_lower) >= len(question_lower) * 0.5:
distance = dice(question_lower, request_lower)
if distance > best_distance:
best_distance = distance
best_match = question
best_index = i
if best_match and best_distance > 0.3: # Threshold for partial match
debug_log(f"DEBUG: Found best partial match at index {best_index} with distance {best_distance:.3f}")
return best_match
debug_log(f"DEBUG: No matching question found for cleaned: {cleaned[:100]}...")
return None
def get_answer(self, question: Dict) -> str:
answer = question["answer"]
if isinstance(answer, str):
normalized = normalize_number(answer)
return str(normalized) if normalized is not None else answer
return str(answer)
class CodeDataset:
"""Stand-in for a code model: replays canonical solutions as chat replies.
Handles both code suites: HumanEval keys off the function stub, ClassEval
off the class skeleton.
"""
DEFAULT_SOURCE = {
"humaneval": "openai/openai_humaneval",
"classeval": "FudanSELab/ClassEval",
}
def __init__(self, dataset_type: str, source: Optional[str] = None):
self.dataset_type = dataset_type
self.source = source or self.DEFAULT_SOURCE[dataset_type]
self.questions: List[Dict] = []
self._load_dataset()
def _load_dataset(self):
print(f"Loading {self.dataset_type} dataset (source: {self.source})...")
local = Path(self.source)
if local.exists():
text = local.read_text()
if local.suffix == ".json":
rows = json.loads(text)
else:
rows = [json.loads(ln) for ln in text.splitlines() if ln.strip()]
else:
import datasets
rows = [dict(r) for r in datasets.load_dataset(self.source, split="test")]
self.questions = rows
print(f"{self.dataset_type} dataset loaded: {len(self.questions)} questions")
def _stub_field(self) -> str:
return "skeleton" if self.dataset_type == "classeval" else "prompt"
def _get_question_text(self, question: Dict) -> str:
return question[self._stub_field()]
def find_question(self, request_text: str) -> Optional[Dict]:
for question in self.questions:
stub = question[self._stub_field()].rstrip()
if stub and stub in request_text:
return question
return None
def get_answer(self, question: Dict) -> str:
if self.dataset_type == "classeval":
return question["solution_code"]
return question["prompt"] + question["canonical_solution"]
class Simulator:
def __init__(
self,
port: int = 8033,
host: str = "localhost",
success_rate: float = 0.8,
dataset_split: str = "train",
dataset_type: str = "aime",
dataset_source: Optional[str] = None
):
self.port = port
self.host = host
self.success_rate = success_rate
self.dataset_type = dataset_type
if dataset_type in ("humaneval", "classeval"):
self.dataset = CodeDataset(dataset_type, dataset_source)
else:
self.dataset = AimeDataset(dataset_split, dataset_type)
self.eval_state = EvalState(
id=dataset_type,
tasks=[dataset_type],
task_states={},
sampling_config={"temperature": 0, "max_tokens": 2048}
)
def _generate_response(
self,
question: Dict,
should_be_correct: bool
) -> Dict:
expected_answer = self.dataset.get_answer(question)
if should_be_correct:
response_text = expected_answer
else:
response_text = self._generate_wrong_answer(question)
if self.dataset_type in ("humaneval", "classeval"):
response_text = (
f"Here is the implementation:\n\n```python\n{response_text}\n```\n\n"
"Let me know if you want it optimised further."
)
comp_tokens = random.randint(10000, 60000)
tps_gen = random.uniform(90.0, 110.0)
t_gen_ms = comp_tokens / tps_gen * 1000
return {
"id": f"chatcmpl-{int(time.time())}",
"object": "chat.completion",
"created": int(time.time()),
"model": "llama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": response_text
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 100,
"completion_tokens": comp_tokens,
"total_tokens": 100 + comp_tokens
},
"timings": {
"predicted_ms": t_gen_ms,
"predicted_per_second": tps_gen
}
}
def _generate_wrong_answer(self, question: Dict) -> str:
expected_answer = self.dataset.get_answer(question)
if self.dataset_type == "classeval":
return re.sub(r"(\n\s+def [^\n]+:\n)", r"\1 return None\n",
question["skeleton"])
if self.dataset_type == "humaneval":
return question["prompt"] + " return None\n"
if expected_answer.isdigit():
wrong_answer = str(int(expected_answer) + 1)
else:
wrong_answer = expected_answer + " (wrong)"
return wrong_answer
def _process_request(self, request_data: Dict) -> Dict:
messages = request_data.get("messages", [])
if not messages:
return {"error": "No messages in request"}
request_text = messages[0].get("content", "")
debug_log(f"DEBUG: Received request with content: {request_text[:150]}...")
question = self.dataset.find_question(request_text)
if not question:
debug_log(f"DEBUG: find_question returned None")
return {"error": "No matching question found"}
should_be_correct = random.random() < self.success_rate
response = self._generate_response(question, should_be_correct)
task_id = "aime"
self.eval_state.task_states[task_id] = {
"correct": should_be_correct,
"expected": self.dataset.get_answer(question),
"predicted": response["choices"][0]["message"]["content"]
}
return response
class RequestHandler(BaseHTTPRequestHandler):
def do_GET(self):
if self.path == "/v1/models":
self._send_json({"data": [{"id": "llama", "object": "model"}]}, 200)
return
self._send_json({"error": "Not found"}, 404)
def do_POST(self):
if self.path != "/v1/chat/completions":
self._send_json({"error": "Not found"}, 404)
return
try:
content_length = int(self.headers.get("Content-Length", 0))
body = self.rfile.read(content_length)
request_data = json.loads(body) if body else None
if not request_data:
self._send_json({"error": "Invalid JSON"}, 400)
return
if simulator is None:
self._send_json({"error": "Simulator not initialized"}, 500)
return
response = simulator._process_request(request_data)
self._send_json(response, 200)
except json.JSONDecodeError:
self._send_json({"error": "Invalid JSON"}, 400)
except Exception as e:
print(f"Error processing request: {e}")
self._send_json({"error": str(e)}, 500)
def _send_json(self, data: dict, status: int = 200):
body = json.dumps(data).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, format, *args):
# Suppress default request logging
pass
def main():
parser = argparse.ArgumentParser(
description="llama-server simulator for testing eval scripts"
)
parser.add_argument(
"--port",
type=int,
default=8033,
help="Server port (default: 8033)"
)
parser.add_argument(
"--host",
type=str,
default="localhost",
help="Server host (default: localhost)"
)
parser.add_argument(
"--success-rate",
type=float,
default=0.8,
help="Success rate 0-1 (default: 0.8)"
)
parser.add_argument(
"--dataset",
type=str,
default="aime",
choices=["aime", "aime2025", "humaneval", "classeval"],
help="Dataset type (default: aime)"
)
parser.add_argument(
"--dataset-split",
type=str,
default="train",
help="AIME dataset split to use (default: train)"
)
parser.add_argument(
"--dataset-source",
type=str,
default=None,
help="Override the dataset location (local .jsonl path or HuggingFace repo id)"
)
args = parser.parse_args()
global simulator
simulator = Simulator(
port=args.port,
host=args.host,
success_rate=args.success_rate,
dataset_split=args.dataset_split,
dataset_type=args.dataset,
dataset_source=args.dataset_source
)
server = HTTPServer((args.host, args.port), RequestHandler)
server_thread = threading.Thread(target=server.serve_forever, daemon=True)
server_thread.start()
print("\n=== llama-server-simulator ===")
print(f"Server running on http://{args.host}:{args.port}")
print(f"Success rate: {args.success_rate}")
print(f"{args.dataset} dataset loaded: {len(simulator.dataset.questions)} questions")
print("\nPress Ctrl+C to stop\n")
try:
server_thread.join()
except KeyboardInterrupt:
print("\nShutting down...")
server.shutdown()
if __name__ == "__main__":
main()
llama.cpp/examples/llama-eval/test-simulator.sh
Executable
+86
View File
@@ -0,0 +1,86 @@
#!/bin/bash
set -e
# Get the directory where this script is located
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
echo "=== llama-server-simulator Test Script ==="
echo ""
PORT=8033
SUCCESS_RATE=0.8
TEST_PORT=8034
echo "Starting simulator on port $PORT with success rate $SUCCESS_RATE..."
source "$SCRIPT_DIR/venv/bin/activate"
python3 "$SCRIPT_DIR/llama-server-simulator.py" --port $PORT --success-rate $SUCCESS_RATE > /tmp/simulator-test.log 2>&1 &
SIMULATOR_PID=$!
echo "Waiting for simulator to start..."
sleep 5
# Helper function to make a request and extract the answer
make_request() {
local question="$1"
curl -s -X POST http://localhost:$PORT/v1/chat/completions \
-H "Content-Type: application/json" \
-d "{
\"model\": \"llama\",
\"messages\": [
{\"role\": \"user\", \"content\": \"$question\"}
],
\"temperature\": 0,
\"max_tokens\": 2048
}" | python3 -c "import sys, json; data = json.load(sys.stdin); print(data.get('choices', [{}])[0].get('message', {}).get('content', data.get('error', 'No response')))"
}
# Test question (repeated in multiple tests)
TEST_QUESTION="Quadratic polynomials P(x) and Q(x) have leading coefficients 2 and -2, respectively. The graphs of both polynomials pass through the two points (16,54) and (20,53). Find P(0) + Q(0)."
echo ""
echo "=== Test 1: Correct Answer ==="
echo "Sending request with known question..."
answer=$(make_request "$TEST_QUESTION")
echo "Answer: $answer"
echo "Expected: 116"
echo "Correct: $([ "$answer" == "116" ] && echo "Yes" || echo "No")"
echo ""
echo "=== Test 2: Wrong Answer ==="
echo "Sending request with known question (success rate 0.0)..."
answer=$(make_request "$TEST_QUESTION")
echo "Answer: $answer"
echo "Expected: 116"
echo "Correct: $([ "$answer" == "116" ] && echo "Yes" || echo "No")"
echo ""
echo "=== Test 3: No Matching Question ==="
echo "Sending request with non-matching text..."
response=$(make_request "What is the capital of France?")
echo "Response: $response"
echo "Expected: No matching question found"
echo "Correct: $([ "$response" == "No matching question found" ] && echo "Yes" || echo "No")"
echo ""
echo "=== Test 4: Success Rate Verification ==="
echo "Sending 10 requests to test success rate..."
correct_count=0
for i in {1..10}; do
answer=$(make_request "$TEST_QUESTION")
if [ "$answer" == "116" ]; then
correct_count=$((correct_count + 1))
fi
echo " Request $i: Answer = $answer"
done
echo "Correct answers: $correct_count/10"
echo "Expected: ~8/10 (80% success rate)"
echo "Success rate: $(echo "scale=1; $correct_count * 10" | bc)%"
echo ""
echo "=== Test Complete ==="
echo "Stopping simulator..."
kill $SIMULATOR_PID 2>/dev/null
wait $SIMULATOR_PID 2>/dev/null || true
echo "Simulator stopped."