Hermes Bench (fork)

This is the YMRTech fork of Bent-Solutions/hermes-bench — a local benchmarking UI for LLMs and AI agents.

Why we forked

We needed Hermes Bench working against our ninfer-3090 backend, which required changes the upstream project doesn't ship:

• llama.cpp backend fixes — auto-detect the model id via /v1/models and honor reasoning_content in responses (upstream assumed a fixed model name and OpenAI-style reasoning fields only).
• Reasoning effort toggle + bounded 503 retry for single-slot backends — ninfer-3090 serves one model at a time, so requests can hit a transient 503; the runner now retries with backoff instead of failing the run.
• Timeout slider fix — the UI timeout control previously didn't apply to benchmark runs; it now does.
• Nix flake — upstream's start.sh creates a venv and npm-installs at runtime, which can't work in the NixOS read-only store. The flake builds the frontend (Vite via buildNpmPackage) and backend (nixpkgs python3Packages) declaratively.

Running with Nix

Requires Nix with flakes enabled.

# Build and run (listens on 0.0.0.0:7420)
nix run github:ymrtech/hermes-bench

Or add it as a flake input (this is how nix-config consumes it):

{
  inputs.hermes-bench.url = "git+https://git.ymrtech.com/ymrtech/hermes-bench";
  # nixpkgs is followed, so store paths match your nixpkgs version.
}

Then reference inputs.hermes-bench.packages.<system>.hermes-bench in your system packages or a service definition.

The launcher cds into ~/.hermes-bench (writable state dir — the SQLite DB lives here) and starts uvicorn on port 7420. Open http://localhost:7420.

Development / checks

nix develop .        # shell with backend + pytest deps, cwd set to backend/
nix flake check      # runs the backend pytest suite
nix build .          # build the package

Environment variables (still honored at runtime):

Variable	Description	Default
HERMES_BENCH_PORT	Backend server port	7420
HERMES_BENCH_HOST	Backend bind address	127.0.0.1
HERMES_BENCH_DB	SQLite database path	./hermes_bench.db
HERMES_HOME	Hermes home directory	~/.hermes

Running without Nix

This fork is maintained Nix-first and we don't support the venv/Docker workflows here. For non-Nix setup (start.sh, manual venv, docker-compose), see the original upstream repo — the app logic is shared, though the ninfer-3090 fixes above are specific to this fork.

License

MIT