FiniBench: Agentic AI Efficiency Benchmark

Why does your self-hosted model feel dumber than the official benchmarks suggest?

We evaluate the combination of LLM × Quantization Level × Harness across a wide range of multi-turn agentic AI tasks. By focusing on end-to-end task completion rates, FiniBench cuts through the advertised FP16 quality and reveals which quantization levels you can run and actually solve problems without collapsing.

Intelligence vs Efficiency

Color By

Task Speed Leaderboard

Comparing execution speed (tasks/hr) across evaluated model configurations.

Intelligence Density Leaderboard

Comparing hardware efficiency (tasks/GB) across evaluated model configurations.

Problem Solving: Tasks Completed

Comparing total tasks completed with full score by each harness (pi vs opencode cli).

Execution Speed: Total Time Taken

Comparing aggregate time spent completing tasks by each harness (lower is faster).

AI Agent Leaderboard

Model Name	Speed	Density	Intelligence	Memory	Harness	Reasoning

Why is FiniBench Different?

Quantization & Agent Loops

KL Divergence Fails to Predict Agentic Performance

Model cards are usually advertised at FP16 quality, but most home labs run models at FP8 or even FP4 quality. Quantization levels that show minor quality loss on single-turn tests often suffer total task failure when subjected to multi-step tool calls.

Key Insight: A Q2 quant should be treated as a separate model from its FP16 equivalent.

Harness Impact

Harness Architecture Significantly Impacts Success

The design of an agent harness—including its prompt structures, tool definitions, and edit formats—exerts a major influence on task completion rates. Benchmark suites must evaluate models across standardized harnesses to isolate weight precision effects from harness overhead.

Key Insight: LLMs don't work alone; the harness and deployment configs matter even more on resource-constrained devices.

Task Efficiency

Token Efficiency and Reasoning Length Confound Results

Evaluating agentic workflows requires shifting from raw generation speed (tokens per second) to outcome-oriented metrics: Task Speed (tasks/hr) and Intelligence Density (tasks/GB). These metrics provide an objective measure of system utility on fixed hardware footprints.

Key Insight: Outcome efficiency measures real work delivered per unit of resources.

Our Methodology

✨ Model Serving

Our model serving infrastructure evaluates open-weight AI models across the two most popular local inference runtimes: llama.cpp and vLLM. All models are served on an Nvidia DGX Spark with 128GB of VRAM.

Weight Precision & Quantization: We test FP16 unquantized baselines alongside FP8, INT4, NVFP4, and GGUF / EXL2 K-quants (Q8_0, Q6_K, Q4_K_M, Q3_K_M, Q2_K).
Context & Memory Allocation: Agentic workflows with reasoning LLMs benefit immensely from long context windows. We default to the model maximum with optional KV-cache quantization enabled.
Speculative Decoding: When possible, we test models with various speculative decoding methods such as MTP, DFlash, or DSpark for the best single-user performance.
Sampling & Reproducibility: Temperature, Top-P, and Top-K are set to model-recommended values. Each test is run 3 times and the results are averaged.

🤖 Harness Setup

Our custom agent harness runs in an isolated container and orchestrates multi-turn tool loops, terminal execution, file system modifications, and error feedback recovery. The harness runs on an Apple M1 Pro MacBook Pro with 32GB of RAM.

Most Popular Agents: We evaluate models across popular agent frameworks (such as OpenCode and Pi) in their out-of-the-box configurations.
Loop Limits: Agents are permitted up to a predefined maximum number of sequential turns per problem instance before triggering a timeout or unrecoverable loop failure.

📝 Benchmark Tasks

We evaluate models across a private test suite of real-world agentic workflows. To prevent benchmark contamination, test prompts remain private, but each test has a descriptive name indicating its scope. Task difficulty varies and is scored accordingly:

Greenfield Software Engineering: Building new features, modular components, and applications from scratch based on user specifications.
Software Maintenance & Bug Fixing: Navigating existing large codebases, diagnosing errors, and applying clean, git-verified multi-file diffs.
Software Porting: Taking exisitng software and porting it to a different language or framework.
Data Analytics & Report Generation: Data scraping, parsing, calculating summary metrics, and synthesizing findings into formatted reports.
Content Translation & Summarization: Multi-language translation, document restructuring, and cross-format content synthesis.
Enterprise Business Operations: ingesting incoming support tickets, performing initial intent classification via lightweight models, querying internal knowledge bases.

📊 Evaluation & Metrics

Tasks are evaluated using a combination of deterministic verification tests and LLM-as-a-judge assessments to measure end-to-end task completion.

Automated Verification: Evaluation combines deterministic checks (e.g., git diff line verification, automated unit tests) and calibrated LLM judges.
Task Speed (Tasks/hr): Quantifies total verified multi-turn tasks completed per 60 minutes of wall-clock execution time.
Intelligence Density (Tasks/GB): Measures model size efficiency by normalizing task throughput against peak run memory consumption.
Trace Logging: Every evaluation run generates an execution trace containing input/output token counts, run memory footprint, context utilization, and complete terminal logs.