There is a small translation problem in asking an assistant about a screen. You can see the thing you mean, but the assistant needs a reference. “The second item below the heading” is precise until the layout changes. Copying text may be awkward or impossible. A screenshot contains the context, but it can still leave the assistant wondering which part matters.
Drawing gives the conversation a shared location. With Surf, the person can circle or draw on screen to ask about something in view. Surf can also draw arrows and highlights back to guide the next step. This is not a new language to learn. It is an everyday gesture applied to the place where the task is already happening.
Pointing supplies context
Imagine a settings screen with several similar rows. The person knows which setting is confusing, but may not know its name. Circling the relevant row narrows the question. “What does this do?” now has a visible anchor. Screen context and the drawn mark work together: one provides the surrounding information, and the other identifies where the person's attention is.
That anchor does not replace the request. A circle around a price could mean explain it, compare it or help buy the item. These are different intentions, with different consequences. Useful assistance connects the mark to the person's words and asks for more detail when needed. Pointing makes a conversation shorter; it should not make an important decision implicit.
A circle says where to look. Your words say what to do.
Answer in the same place
Some answers are easier to use when they are visible on the screen. If the question is where to save a change, an arrow can connect the explanation to the relevant control. A highlight can draw attention to the part of a form that needs review. The person does not have to keep translating a paragraph back into the layout in front of them.
Surf's ability to draw on screen makes that exchange two-way. The person indicates where help is needed; Surf can indicate the next step. The aim is guidance that stays close to the task. A visual mark should have a clear job, rather than becoming decoration that competes with the app's own information or obscures a choice the person needs to read.
Keep the original app in view
The overlay is important to this interaction. Surf lives on top of the Android app the person is using, so asking for help does not necessarily require a trip to another screen. The question, the object and the answer can remain in the same context. That can be especially useful for a task whose difficulty is finding the next control rather than understanding a long explanation.
The product artwork on this site shows examples of guidance and a calendar change. These are illustrations of the experience, not recordings of a live task running in the browser. The distinction matters. A website can demonstrate how a request is framed, while the real app must work with the current screen and the permissions the person has granted.
Access still has a boundary
Surf uses Accessibility to read and act on screen content. Drawing is a way to direct the request, not a way around the need for permission. The person can control Accessibility access in Android Settings. A circle does not grant access to a connected account, authenticate a session or authorise an unrelated action. Those boundaries remain separate from the visual gesture.
Secure input remains separate too. Passwords and OTPs are never sent to the model. People should not be encouraged to circle a secret as a shortcut for sharing it with an assistant. The useful context is the task and the place where help is needed. A secure value belongs in its secure input flow, even when the rest of the conversation is visual.
A mark is not permission to spend
Pointing at a product can start a useful conversation about an order. It does not approve the final payment. Surf asks before anything that spends money and shows a pay card for review. The person still needs to see the merchant, items and total and decide whether to proceed. A quick gesture at the start cannot settle a checkout's final details.
This is a helpful general rule for designing direct interactions. Making an input easier should not make its meaning broader than the person expects. A circle is excellent at identifying a region. It is less good at expressing every condition attached to a purchase or booking. The surrounding conversation and the approval step carry the decisions that the gesture cannot safely communicate on its own.
Useful for the small stuck moments
Draw-to-ask is most interesting in ordinary situations: a control that is hard to find, an unfamiliar option, a detail that needs a second look. The person does not need to describe the whole app or start a formal troubleshooting session. They can point to the place where the task became confusing and explain what they want to achieve.
This draft explores why that interaction belongs in Surf alongside text and voice. Each input gives the person a different way to make their intention clear. The best choice depends on the moment. Sometimes a sentence is enough. Sometimes speaking is easier. And sometimes the clearest way to begin is to draw one circle and ask, ?What about this??
Draft for review. Examples are illustrative; no metrics, customer quotations or research results are claimed.
Back to the notebook ↗