Can a Local AI Model Choose the Right Workflow? Our Laya Pilot

TL;DR

Can a local model select a useful workflow route? Our Laya pilot explored that question in shadow mode: it recorded advice while the existing agent route remained in control. It did not establish better routing accuracy, lower cost or improved task quality. The useful result is a clearer evaluation method, separating advice, execution and accepted outcomes.

What Laya Is, Before the Pilot

A local typed-decision model returns a bounded result, such as a choice from an allowed set, a score or a yes/no answer. That shape is easier to validate and log than unrestricted prose. Local execution can avoid sending the decision request to an external model service, but it does not automatically make the surrounding workflow private or faster. Those properties depend on the implementation and require their own checks.

Shadow Advice vs. Executed Decisions: What Arm B Did

Think of it like a co-pilot calling out a turn while someone else keeps both hands on the wheel. The suggestion is logged. It is not followed.

In the pilot, Arm A is the existing agent route, the one that runs tasks today. Arm B is Laya's recommendation of a content/general/none workflow, generated in shadow mode only. Per the decision-pilot documentation, Laya's advice in this pilot never authorized publication, triggered a tool call, bypassed a safety check, or caused a deployment. It was recorded alongside Arm A's actual route so the two could later be compared, but comparison is not the same as control. Nothing in this pilot let Laya's opinion change what happened.

Reading the Numbers: Raw Events Are Not Independent Tasks

An event can be one step inside a task. A log containing many events may represent far fewer independent decisions, and fewer still with comparable routes and usable outcome labels.

The October 6 pilot snapshot separated current-source, historical and missing-context events. The source revision matters: a routing rule changed halfway through a study cannot be pooled with an earlier version without explanation. Some tasks had comparable routes and others did not. Differences between the two routes were candidates for review, not evidence that either route was correct. Exact aggregate counts are omitted from this refresh because we could not re-bind them to the frozen snapshot.

Table 1: Pilot Scale at a Glance

Historical and missing-context events remain visible in the report and are intentionally not pooled with current decisions.

AI-Reviewed Labels Are Not Human Ground Truth

An AI-reviewed label means a model judged the outcome, not a person. It's a useful, fast signal for development work. It is not a validated ground-truth benchmark.

The pilot distinguishes human-adjudicated labels from AI-reviewed labels. A model can help organize a review queue, but its judgment is not automatically ground truth. Labels also need balanced coverage: a large collection dominated by one route may say little about a rare but consequential choice. Record unresolved cases and missing context rather than silently excluding them from the headline result.

Taxonomy: Family, Phase, Risk, and Why a Family Doesn't Pick the Route

Knowing a task's "family" tells you what kind of work it is, not which workflow route handled it. Those are recorded as separate facts, on purpose.

The pilot introduces a finer-grained workflow taxonomy with nine families (feature delivery, defect resolution, code health, platform operations, content/growth, research/analysis, knowledge management, coordination, and direct response), plus an immediate phase, the affected areas, and a risk stratum. All of that is recorded separately from the content/general/none route choice. That separation matters: a "feature delivery" task doesn't automatically route to "general," and a "defect resolution" task doesn't automatically route to "none." The taxonomy gives reviewers a richer description of what a task is; it doesn't dictate or explain why Arm A or Arm B picked the route it did.

Check Passed, Task Accepted: Two Different Things

A command finishing without error is like a single part passing its own certification. It tells you that one piece worked, not that the whole assembly was accepted into the final build.

A command finishing without error is evidence that the command ran cleanly. It does not show that a person accepted the resulting task. A useful pilot records both: the check result and the decision about whether the output met the brief. A missing acceptance record is unknown, not a success inferred from a green command.

What We Can Claim Today

The supportable, positive finding from this round is about measurement quality, not outcomes: the pilot now captures tasks with more context, separates consultation evidence from route-match evidence from execution evidence, produces tighter review packets for human reviewers, applies a finer taxonomy, and records explicit abstentions instead of silently dropping unclear cases. That is real progress in visibility.

This design does not support a causal improvement in quality, cost or speed, or a claim that Laya routes more accurately than the existing agent. A convincing comparison needs enough independently labeled, source-matched tasks across the relevant families and risks, followed by execution and outcome measurement under a controlled study.

The Next Experiment, in Plain English

The honest next step is a proper, controlled comparison, not a bigger pile of shadow logs. That means balanced, source-matched labels instead of the current lopsided mix; a closed holdout set that isn't touched during analysis; and a run where Arm B is selected and followed on matched real tasks, under controlled conditions, rather than only logged in the background. That run should record whether the resulting task was accepted, how much rework it needed, how long it took, and whether it involved complete coding-agent usage. That is the concrete path from "we can see more clearly" to "we have evidence of improvement."

Table 2: What Would Change Between Now and a Research-Grade Study

Sources and Further Reading

This article describes the October 6 shadow-pilot design and its evidence limits. The current repository documents later changes separately; it should not be read as proof that a later installation behaves identically. No private task transcripts or credentials are included. The implementation references below describe the boundaries that an evaluator should inspect.

• Laya upstream model reference: consult the upstream link recorded in the typed-decision documentation for the exact model used in the pilot.
• Typed decision layer: https://github.com/Rhize-Media/rhize-plugins/blob/main/docs/typed-decision-layer.md
• Pilot design and label boundaries are documented with the typed decision layer. Check the source revision and operating mode before applying its findings to a current installation.
• When evaluating your own workflow, record the task family, the route selected, the action executed and the accepted outcome as separate fields.

Before adding a routing model to a real workflow, define the decision it may advise on, the decisions it cannot authorize and the evidence that would justify following its advice. Browse the Rhize plugin documentation for the implementation and evaluation boundaries.