Local AI systems: harnesses, Argon, and why the benchmarks lieDraft
Running models locally is the easy part. The harness around them is the system. Notes on building one, where Gemini 4 Argon fits, and a critique of the benchmarks everyone quotes.
A model is a function. A system is the thing that decides when to call it, with what, how to check the answer, and what to do when it is wrong. Most of the interesting engineering in “local AI” is in the second part, and almost none of the public discussion is.
Part 1 — What “local” actually buys you
Three things, and only the first is about privacy:
- Data boundary. Trade data, positions, and risk numbers do not leave the box. Non-negotiable in finance; nice-to-have elsewhere.
- Latency floor. No network, no queue, no rate limit. A 7B-class model on a workstation GPU answers in the time a hosted frontier model spends in TLS handshakes.
- Determinism you can pin. Same weights, same quantisation, same sampler, same seed → same output. Hosted APIs change under you; a local model is a dependency with a hash.
What it does not buy you is frontier-level reasoning. The honest architecture is tiered: a local model for the 90% of calls that are extraction, routing, classification, and summarisation; a hosted frontier model for the 10% that need it, with the data boundary enforced by the harness, not by hope.
Part 2 — The harness is the product
A harness is the runtime around the model. Mine has five layers; each one exists because of a failure I hit without it.
| Layer | Job | Failure it prevents |
|---|---|---|
| Router | Pick model + tier per request | Paying frontier prices for "yes"/"no" |
| Context builder | Assemble prompt from typed sources (files, DB rows, tool results) | Prompt drift; untraceable answers |
| Tool bus | Typed, sandboxed functions; every call logged | Silent side effects |
| Verifier | Schema checks, unit tests, a second model, or a human | Confident garbage |
| Ledger | Append-only log of every prompt, completion, tool call, cost | “Why did it do that?” |
The ledger is the piece people skip and the piece that makes the rest debuggable. It is also what lets you replay a session against a new model — which is the only evaluation that matters (see Part 4).
Part 3 — Where Gemini 4 Argon fits
Argon is the model I currently route the hosted tier to. Concretely, the use cases where it earned the slot:
- Long-context reconciliation. Feeding an entire day’s order log plus the risk report and asking for inconsistencies. Local models lose the thread; Argon doesn’t.
- Code review of generated kernels. The local model writes the CUDA; Argon reviews it against the spec and the benchmark output. Cheaper than a human first pass, and it catches the race conditions humans skim past.
- Structured extraction from PDFs — term sheets, prospectuses — where layout matters and OCR alone is not enough.
Where it does not earn the slot: anything latency-bound, anything with position data, anything a 7B model gets right 98% of the time.
Part 4 — A critique of the benchmarks
Every model launch comes with a table. Here is why I stopped reading the table.
Contamination is the default, not the exception. Public benchmarks are on the internet; the internet is the training set. A score on a benchmark that predates the model’s training cutoff measures memorisation plus reasoning, in unknown proportion.
Pass@1 on a static set is the wrong unit. My workload is not “answer this question once”. It is “run this 40-step tool loop and don’t break anything”. A model that scores 3 points higher on a QA benchmark but loses the thread at step 25 is a worse model for me.
Aggregates hide the distribution. Two models at 85% can differ completely in which 15% they fail. If the failures correlate with my domain (numerics, finance jargon, long tables) the headline number is noise.
Elo arenas measure preference, not correctness. Human raters reward confident, well-formatted answers. That is exactly the failure mode a verifier layer exists to catch.
Benchmarks don’t price the harness. A model that needs three retries and a verifier to hit 95% costs three times what the table implies. Cost per verified task is the number; nobody publishes it.
What I do instead
Replay. The ledger from Part 2 holds a few thousand real sessions with known-good outcomes. A new model is evaluated by replaying them and diffing: tool calls made, final outputs, tokens spent, wall time. It is a private benchmark that cannot be contaminated, measures my distribution, and prices the whole loop.
That is also the honest answer to “which model is best”: the one that wins the replay this month, on your ledger. Anything else is marketing.