Building an Intent-Driven LLM Router: A Local AI Project Powered by Ollama

If you’ve spent any real time working with LLMs, you’ve probably run into the same wasteful habit I did: sending every single query to one giant, general-purpose model. Asking “what time is it in Tokyo” to a 30-billion-parameter reasoning model is like calling a structural engineer to hang a picture frame. It works, but it’s absurd overkill — and it’s expensive, slow, and often no better than a much smaller, specialized model would be.

That frustration is exactly why I built Intent-Driven LLM Orchestrator, a fully local AI project that runs entirely on your own machine through Ollama — no API keys, no billing, no cloud inference. The only thing that ever leaves your machine is a DuckDuckGo search call made by the agent itself. Everything else — routing, generation, and even quality control — happens on your own hardware.

In this post, I’ll walk through how the system thinks, why I designed it the way I did, and what building this LLM project taught me as an AI data engineer. I’ll also share the repo at the end if you want to run it yourself.

The Problem With Sending Every Query to One Giant Model

Modern LLM deployments tend to fall into one of two traps:

• One model does everything. Simple, but wasteful — you’re paying full reasoning-model cost for trivial requests, and you’re stuck with mediocre code generation from a model that was never specialized for it.
• Manual model selection. The user (or the frontend) has to decide which model to call. That’s a poor experience and doesn’t scale.

What I wanted instead was a system that decides for itself which kind of intelligence a request actually needs, and only pays for that. That’s the entire premise behind this ai project: routing is a first-class architectural decision, not an afterthought.

What Is Intent-Driven Routing?

At the front of the pipeline sits a small, fast classifier — just mistral:7b — whose only job is to read the incoming query and output a single token: the name of the model that should handle it. It never answers the query itself, and it never explains its reasoning. If it returns anything outside the expected set of model names, the system raises an error rather than silently guessing.

Because the classifier’s job is so narrow, it’s cheap to run on every single request — which is the whole point. You spend a tiny amount of compute up front to avoid wasting a lot of compute later.

model = LLMFactory.get_model(profile="coding")

That one line is all any part of the system needs to write to get a correctly configured model. Every profile — temperature, context window, reasoning mode, memory residency — is declared once in a config dictionary and resolved by a single factory class. Swapping a model or tuning a parameter never touches business logic; it’s a one-line change in one place.

Architecture: How Requests Flow Through the System

Once the classifier decides, the query is handed to one of three specialists, each tuned very differently:

Specialist	Role	Personality
Agent model	Live data, exact math, current date/time	Low temperature, has real tools
Chat model	Conversation, writing, brainstorming	Largest model, highest temperature, reasoning on
Coding model	Generation, debugging, refactoring	Long context window, low temperature

After a specialist produces an answer, it doesn’t go straight back to the user. It passes through an independent LLM-as-a-judge, which scores the response and decides whether it’s actually good enough to show. Only on a passing verdict does the answer get returned as-is; a failing verdict triggers a single bounded retry.

This judge-and-retry loop is, in my opinion, the most underrated part of this llm project. Most hobby chatbots trust whatever the model says. This one checks its own work first.

Meet the Specialists

The agent is the only specialist with real tools: web search, a calculator for exact arithmetic, and a timezone-aware clock. Its system prompt draws a hard line — a tool is required for anything current, live, externally changing, or exactly computable, and it treats every tool result as untrusted data rather than as an instruction. That last rule matters more than it sounds: it’s a direct defense against prompt injection sneaking in through search results.

The chat model runs on a sliding conversational memory window and is instructed to stay concise, avoid fabricating facts, and respect whatever tone or format the user actually asked for.

The coding model gets a much longer memory window, because code conversations reference earlier context far more than casual chat does. Its prompt explicitly bans the classic LLM cop-out of writing // rest of the code here instead of finishing the implementation — a small but very practical guardrail that saves a lot of back-and-forth.

One detail I’m particularly happy with: the classifier prompt explicitly states that words like “write,” “create,” and “generate” don’t automatically mean “send this to the coding model.” “Write a poem” goes to chat. “Write a parser” goes to coding. Getting that distinction wrong is one of the most common failure modes in naive intent classifiers, so I built a test suite around exactly these near-collisions.

The LLM Judge: Quality Control Without a Human in the Loop

Instead of letting the judge return free-text opinions, it’s forced to return a validated, structured object:

class LLMJudgeResponse(BaseModel):
    score: float # 0-10, overall quality
    correctness: float # 0-10
    relevance: float # 0-10
    completeness: float # 0-10
    instruction_following: float # 0-10
    verdict: Literal["PASS", "FAIL"]
    critique: str

That schema is generated and enforced through a Pydantic output parser, so a malformed judge response fails loudly instead of quietly poisoning the pipeline. This wasn’t just a nice-to-have — early in development, without structured output, the “judge” occasionally decided to answer the original question itself instead of scoring the answer. Locking the output shape down solved that immediately.

I deliberately made the judge fully deterministic (temperature 0.0) while the chat model gets real creative freedom (0.6). If you’re grading answers, you don’t want the grader to be creative too.

What Building This Taught Me as an AI Data Engineer

A few lessons stood out more than I expected going in:

• Structured output is non-negotiable for anything downstream. The moment an LLM’s output feeds into code rather than a human’s eyeballs, free text becomes a liability. Pydantic schemas + output parsers turned “hope it formats correctly” into “guaranteed shape or a loud failure.”
• Negative prompt constraints matter as much as positive ones. Telling a classifier what a word doesn’t mean (“write” ≠ code) fixed more misroutes than any amount of positive examples did.
• Lazy initialization is cheap insurance. A user who never asks a coding question never pays the cost of loading a 30B coding model into memory. That single design choice makes the whole system far more practical to run on consumer hardware.
• An LLM judge changes the failure mode from silent to visible. Instead of a wrong answer slipping through unnoticed, you get a scorecard and a critique explaining exactly why it failed — which is a much better place to debug from.

If you work anywhere near production LLM systems, none of this will feel exotic — but seeing it all land in one small, runnable ai project made the trade-offs a lot more concrete for me than reading about them ever did.

Try It Yourself

The whole thing runs locally through Ollama. At a high level, getting started looks like this:

bash
ollama pull mistral:7b # intent classifier
ollama pull qwen3:8b # agent
ollama pull qwen3:30b # chat
ollama pull qwen3-coder:30b # coding
ollama pull gpt-oss:20b # judge

cd src
python main.py

You don’t need every model pulled to get started — mistral:7b is mandatory since nothing routes without it, but you can bring in only the specialists you actually plan to use.

I’ve kept the full source, the prompts’ design rationale, a test harness with deliberately tricky “near-collision” queries, and a documented list of known limitations (yes, I’m upfront about the rough edges) on GitHub:

Repo: intent-driven-llm-orchestrator

Final Thoughts

This project started as a way to stop wasting compute on trivial queries, but it turned into a much more useful exercise in LLM orchestration, structured evaluation, and self-correction loops — the kind of problems that show up constantly once you move past toy chatbots and start building anything closer to a real system. Whether you’re an AI data engineer, a hobbyist experimenting with Ollama, or just curious how a multi-model router is actually wired together, I’d encourage you to clone it, break it, and see what you’d do differently.

If you build on top of it or spot something worth fixing, I’d genuinely like to hear about it — that’s half the fun of putting an llm project out in the open.