FiniBench: Agentic AI Efficiency Benchmark
We evaluate the combination of LLM × Quantization Level × Harness across a wide range of multi-turn agentic AI tasks. By focusing on end-to-end task completion rates, FiniBench cuts through the advertised FP16 quality and reveal which quantization actually solves problems without collapsing.
AI Benchmark Visualizations
Quantization Degradation in Multi-Turn Agentic AI Loops
Comparing single-turn AI benchmark accuracy (MMLU, HumanEval) against multi-turn agent task completion (%) across GGUF & EXL2 weight precision levels (FP16, Q8_0, Q4_K_M, Q2_K, IQ2_XXS).
Harness Architecture Impact on AI Task Completion
Evaluating AI agent task completion rate per hour (CR/hr) and success (%) across standardized edit harnesses (Aider, Claude Code, Cline, SWE-bench).
Deployment Efficiency: CR/hr vs Memory Footprint (GB)
Evaluating local AI completion velocity (CR/hr) relative to VRAM allocation on runtimes like llama.cpp, Ollama, and vLLM. Bubble size indicates agentic success rate.
AI Model Leaderboard
| Model Name | Status | Quant | CR / hr | CR / GB | Agent Success | Single-Turn Acc | VRAM Footprint | Raw Speed |
|---|
Why is FiniBench Different?
KL Divergence Fails to Predict Agentic Performance
Model cards are usually advertised at FP16 quality, but most home labs run models at FP8 or even FP4 quality. Quantization levels that show minor quality loss on single-turn tests often suffer total task failure when subjected to multi-step tool calls.
Harness Architecture Significantly Impacts Success
The design of an agent harness—including its prompt structures, tool definitions, and edit formats—exerts a major influence on task completion rates. Benchmark suites must evaluate models across standardized harnesses to isolate weight precision effects from harness overhead.
Token Efficiency and Reasoning Length Confound Results
Evaluating agentic workflow requires shifting from raw generation speed (tokens per second) to outcome-oriented metrics: Completion Rate per Hour and Completion Rate per Gigabyte. These metrics provide an objective measure of system utility on fixed hardware footprints.