When a new AI model arrives, I want to give it something more useful than my attention.
I want to give it a job interview.
The demonstrations can be extraordinary. But my application has a case to route, facts to extract or a proposal to check. A model earns its place by doing that work well.
That is the premise of my Adaptive AI Model Routing Engine. The application supplies a role, a prompt and its priorities. The engine chooses among permitted candidates using evidence about quality, speed and cost.
Give the work a name
Consider an invented case: an incoming message needs to reach the right workflow. The application knows the destinations, supplies the context and can establish through review whether the routing was right.
The useful question is what this role must return, and what makes the answer correct. In Signal Flow, I make several kinds of work explicit.
Transcriber
Turn speech into text
Write down a voice note, ready for the Extractor to turn it into a structured task draft.
Sentinel
Look for instructions that do not belong
Check incoming material for attempts to steer the system away from its task. Flag suspicious content for the application to handle.
Router
Find where the case belongs
Read the case and work out where it belongs: the workflow, the lane or the intent. Each of those questions is its own role, judged on its own evidence.
Extractor
Find the facts that matter
Pull out the facts the application needs. Keep them grounded in the source, and leave missing information empty.
Normalizer
Give the facts a common form
Turn different ways of saying the same thing into a consistent form, without changing the meaning.
Splitter
Find the separate jobs
Notice when one arrival contains several concerns, and keep each case connected to the material it came from.
Decision proposer
Work out what to do next
Turn the case into proposed actions, with the information needed to carry them out.
Critic
Give the proposal a second look
Find problems in a proposal and explain them clearly enough for a reviewer to check.
Repairer
Put the proposal right
Fix the identified problem, keep what already works, and send the proposal through another check.
A good extractor is not automatically a good critic. Each role needs evidence for its own work. The role stays useful while the candidate models change.
“Best” needs a set of priorities
A person waiting for a result needs a different balance from a background queue. Some work can wait. Some mistakes are expensive.
The application declares a role and a priority, and can name a registered request profile; otherwise the role’s default profile applies. The profile defines the balance of quality, speed and cost; these are inspectable policies.
I decide which models are permitted to compete. Within that permitted pool and the request’s constraints, the engine allocates work. The newest model has no special entitlement.
The job can outlast the model
Candidates can run through different cloud endpoints, including self-hosted infrastructure. That is a deployment distinction. It does not change what the role asks them to do.
For an ordinary unpinned request, the application can leave the model choice to the engine. The engine’s execution workers carry out the selected call and return its result.
An application can also request a group of answers. The architecture needs to preserve which model produced each one.
A test gets the interview started
A new candidate starts with gold sets: fixed examples, known expected outcomes and a grading rule appropriate to the role. They support qualification and regression checks.
An approved candidate can serve on this evidence while live judgments are scarce. That basis stays visible.
Trustworthy live results gradually take over. If they fall clearly below the test’s promise, a regression guard suspends the test evidence. Once enough live evidence has taken over, the test contribution retires and never returns.
What if the right answer lost the vote?
In our invented routing case, three models answer. Two say repairs. One says billing. The application follows the majority; a human later confirms billing.
Giving all three models the majority’s outcome rewards the wrong two and penalises the useful dissenter. Each answer needs its own grade.
The application owns the meaning of success. The engine owns how that evidence affects model selection.
The grader compares each answer with reviewed truth and preserves its execution identity. Neither agreement nor a model’s confidence establishes correctness.
Automatic acceptance alone earns no quality credit. Human judgments provide the reference. AI judgments gain weight only after enough human audits establish their agreement; until then, their quality weight is zero.
A correction can also introduce information the models never saw. The earlier comparison may need to be withdrawn or left unassessed. A changed answer is not automatically evidence of a poor model.
The verdict must reach the next choice
A judgment must survive a restart. A retry must not count it twice. A correction of equal or higher authority must replace what it invalidates; automation cannot overrule a human. Withdrawal must not quietly restore an older judgment.
The consumer records the verdict durably and reports the individual grades. The engine maintains the current assessment and recomputes the evidence. On a role switched to live selection, that evidence feeds subsequent choices.
My test follows the whole path: an application verdict, per-answer grades, changed evidence and a later decision influenced by it. A dashboard number alone does not demonstrate adaptation.
Let a challenger learn something
Routing everything to the incumbent can keep a newcomer untested indefinitely. The engine reserves a bounded exploration budget.
Where reviewed truth can judge an unseen answer, an approved challenger can run in shadow. Its answer is evaluated; it never becomes the application’s answer or triggers an action. Evidence can support admission, but serving still needs approval.
Tool use needs a world to act on. The serving model handles the real request with real tools. I record that execution as the baseline for comparison. Each challenger replays the task with simulated tools and its own snapshot of the starting state. The application keeps the live result; the replay cannot change production.
The simulated tools respond to the calls the challenger chooses, so it can explore a different path through the task. When the captured world cannot support that path, the result stays unassessed.
I compare the challenger’s calls, simulated effects and final answer with the recorded execution. Reviewed outcomes establish which answers were useful or correct; matching the serving model is no guarantee of either. I reuse the live execution, so the extra model work goes into evaluating challengers.
Separate gold tasks test general tool-handling capability using anonymized, app-neutral examples, synthetic records and simulated updates. Those scores stay separate from evidence for a production role. Challenger replays also remain identifiable as simulation evidence, distinct from observed production outcomes.
Evidence from copied tasks can earn only a bounded, approved trial with read-only tools and human review. Those human rulings supply real outcome evidence; ordinary serving still requires a separate qualification once the trial has passed. Trials that execute write tools need their own per-effect safeguards, and roles that produce artifacts such as generated documents stay outside this mechanism.
An answer has more than one ingredient
I keep four concerns conceptually distinct: role instructions, domain memory, case context and the case itself.
A capable model can still fail with wrong memory or missing context. Changing an input changes the experiment.
Evidence therefore stays associated with relevant prompt, memory and context versions, and execution conditions. A new version can inherit a bounded prior from compatible evidence in its own lineage; its own results remain identifiable.
That helps me investigate what changed. It does not establish the cause automatically. Improving domain memory is a learning problem of its own.
More traffic is not the same as more truth
Review only the difficult cases and the evidence describes that population. Reviewers can share blind spots. Narrow roles can stay short of data.
Conservative estimates, coverage checks and audits help; a formula’s confidence level is no guarantee for this changing, selectively reviewed system. Answer quality and the share of work a model can answer need separate attention.
A fixed model can also be justified by a required capability, a reproducible experiment or too little trustworthy feedback.
This remains research and development. Finding a generous grader, an overbroad role or a correction that breaks the loop is useful work. Those findings improve the architecture.
Give the models something worth competing for
I want a better candidate to earn work. I also want to keep a less fashionable model when it continues to do the job well.
The stable part is the responsibility, its constraints and the way we judge its outcome. I can explain what was requested, why the engine chose a candidate and where to investigate when the choice stops working.
A new release gives me another candidate.
The work, and the evidence, decide what it earns.
