The missing layer in AI agents is feedback.

Reliable automation begins with a deceptively simple question: what actually happened?

A command returning successfully is not the same as a task succeeding. A click may land while a screen is changing. A text field may update after a verification check. A connection may disappear after the device has already received the action. These are ordinary system behaviors, but they become expensive when an agent mistakes them for certainty.

Observe before deciding

Our tablet controller keeps a warm connection to the device and reads its accessibility tree. That gives the agent a compact description of the current interface: labels, controls, and a reference to the observed state. The important design choice is that a decision belongs to an observation.

If the interface changes before the action, the controller should return the new state and ask the agent to choose again. It should not silently reinterpret an old element number against a new screen.

An action is a hypothesis. The next observation is its test.

Make uncertainty a first-class result

Consider a dropped connection after a request has started. Automatically replaying the request can duplicate a send or a purchase. The correct result is often neither success nor failure. It is an unknown outcome that needs inspection.

observe → choose → check state → act → observe

stale screen    → choose again
unknown result  → inspect before retrying

Give the model less to misunderstand

In one home-screen comparison, collapsing duplicate labels and removing coordinate padding reduced our controller’s output from 5,037 to 1,006 characters. That was an 80% character reduction on a single screen, not a measured token reduction across every task.

Compact output is valuable when it preserves the distinctions that affect the next decision. Two identical labels at different locations still need separate references. Hidden rows need an explicit omission count. Shorter is only better when it remains honest.

The boundary of the guarantee

An accessibility tree cannot reveal every visual change. The interface can still change between a check and a tap. A robust system documents that race, uses screenshots when necessary, and stops short of treating a single successful command as proof of an entire workflow.

This is the feedback layer we want to build around: small observations, bounded actions, and explicit uncertainty. Intelligence becomes useful when the surrounding system makes its consequences legible.