Designing a Decision Layer for AI Agents
ToolRoute is a live AI model routing platform that helps agents and developers automatically select the best model and MCP server for every task. GPT-4o quality at 10-40x lower cost, proven across 132 real benchmark runs.
- Role
- Founder, Product Designer, Systems Architect
- Stage
- Live product with real telemetry
- Domain
- AI Infrastructure, Developer Tools, Agentic Systems
0
Losses vs GPT-4o
10-40x
Cost savings
132
Benchmark runs
100%
Free to use
The Problem
AI agents are failing because they choose the wrong tools
As the number of LLMs, MCP servers, APIs, and agent frameworks grows, the tool selection problem becomes harder. Developers make routing decisions based on incomplete benchmarks, anecdotal recommendations, or trial and error. Agents make them with even less context.
The result is wasted cost, unreliable outputs, and workflows that have no feedback loop to improve over time. A team running GPT-4o on every task when haiku-3.5 would do the job is paying 10-40x more for the same outcome. A team with no fallback strategy gets silent failures.
The deeper problem is that static benchmarks do not map to real workflow outcomes. What works in a blog post evaluation does not always work when an agent is running live tasks at scale.
Design Challenge
The user is not only human
Most product interfaces are designed for humans. ToolRoute required a different frame entirely: what does it mean to design for both a developer and an AI agent at the same time?
Humans need clear navigation, understandable rankings, trustworthy recommendations, and a way to evaluate trade-offs before committing. Agents need structured signals they can consume programmatically: a tier, a cost estimate, a fallback chain, a confidence score, and a response in under 50ms.
That dual-user model shaped every product decision: the information architecture, the API response shape, the leaderboard design, the routing demo, and the way trust signals are surfaced. The interface had to work for a developer evaluating options and for an agent calling the endpoint in production.
Product Experience
From directory to decision system
ToolRoute is not a static list of models and tools. Every surface was designed to get smarter the more it is used. The product combines discovery, comparison, routing, and telemetry into one compounding system.
Model Routing
Six tiers, 20+ models, cost estimates, and fallback chains. The routing layer stops teams from paying GPT-4o prices for tasks that haiku-3.5 handles with equal quality.
Tool Routing
MCP server recommendations scored from real execution runs, not star counts or documentation quality. Confidence scores reflect actual outcomes.
Automatic Fallbacks
When a model fails or a tool times out, ToolRoute tells the agent exactly what to try next. Fallback chains are part of every recommendation.
Live Leaderboards
Rankings that update from real agent telemetry. What works in a benchmark evaluation and what works in production are often different things. ToolRoute tracks both.
Challenges
Agents compete on real tasks, picking their own models and tools. The competition creates a high-quality stream of comparative evaluation data that improves routing for everyone.
Contribution Economy
Agents that report outcomes earn credits and improve future recommendations. Every run makes the system smarter. Telemetry turns ToolRoute from a directory into a living decision layer.
Trust System
Five dimensions. One value score.
Rankings based on popularity, star counts, or documentation quality are not useful for routing decisions. ToolRoute normalizes every run into five core scores derived from actual execution data.
The weighting reflects what actually matters in production: output quality carries the most weight because a fast, cheap answer that is wrong is worthless. Trust carries the least because it is a threshold condition, not a differentiator.
Value Formula
Score = 0.35 Quality
+ 0.25 Reliability
+ 0.15 Efficiency
+ 0.15 Cost
+ 0.10 Trust
Output Quality
35%Task completion, relevance, and correction burden from real runs.
Reliability
25%Success rate, retry rate, and latency stability over time.
Efficiency
15%Latency, tool-call count, and context overhead per task.
Cost
15%Real economic burden: cost per task and cost per outcome.
Trust
10%Permission scope, auth clarity, and security signals.
Telemetry Loop
Every run makes routing smarter
The contribution flywheel is what separates ToolRoute from a static directory. When agents report outcomes after each run, that data improves recommendations for every other agent using the system.
The flywheel has four levels, weighted by the quality of the contribution: a single run report earns 1x weight, a fallback chain report earns 1.5x, a head-to-head comparative evaluation earns 2.5x, and a full benchmark package earns 4x. Higher-quality evidence creates stronger routing intelligence and earns more credits.
This is the compounding moat. The more agents use ToolRoute, the more accurate the recommendations become. Real execution data beats blog post benchmarks at every scale.
Live Product
Built, shipped, and running in production
ToolRoute is not a concept or a case study prototype. It is a live product with a real API, real benchmark data, and real agents routing through it. Every design decision described here is reflected in what you can see at toolroute.io today.
As Founder, Product Designer, and Systems Architect, I owned the full product direction: positioning, information architecture, routing logic, recommendation UX, trust signals, telemetry loop, and the contribution economy model.
Reflection
What this project proves
ToolRoute shows how I approach emerging product spaces: identify the underlying system problem, turn technical ambiguity into a clear product model, define the right user layers, and design an experience that can grow from MVP into infrastructure.
It also reflects a core belief about where AI product design is going. The best AI experiences will not be defined only by model quality. They will be defined by routing, context, trust, feedback loops, and the system's ability to learn from outcomes. ToolRoute is built around that premise from the ground up.
Designing for a non-human primary user is genuinely new territory. It requires thinking about information architecture, trust signals, and interaction models in ways that standard UX patterns do not cover. That is what made this project interesting to build and what makes it a strong signal for anyone hiring for AI product roles.
Next Case Study
ReviewsRight