Tiko
A buddy next to your cursor that looks, talks and points — measured, not assumed · 2026
Hold a shortcut on a Mac, ask out loud, and it looks at the screen, answers
by voice and flies over to point at the button you need. A rebuild of Clicky on a stack
that costs nothing to run — Gemini’s free tier, speech recognition on the Mac,
the system voices — which moves the hard part to where a free model is weakest:
saying exactly where something is. So pointing is measured rather than assumed. A
benchmark draws six Mac screens in code, records the rectangle of every button, and asks
twenty-nine Hinglish questions through the app’s own path. Its first run found the
parser silently dropping [POINT:185,119,filter button] — a comma where
the prompt asked for a colon — which accounted for every answer it scored as
“no point”, in the benchmark and in the app. The next found the model measuring
on the cursor close-up instead of the screen; labelling that image with the part of the
screen it shows lifted the first guess from 55% to 81%. Now, when the model names a label,
the Mac reads the screen itself and snaps to that text — 27 of 27 on target —
and only icons and switches get a second request: 48 of 58 with 1.53 requests a question
instead of 2. 93 checks. A typed question has gone through the running app to an
answer; nobody has spoken to it yet.
built withswift · swiftui · appkit · screencapturekit · speech · avfoundation · gemini
repository↗
case study — how it works, and what the benchmark showed
Why build it again
Clicky, by Farza Majeed, showed the interaction: an assistant that shares your screen
and points instead of describing. Its own issue tracker listed what it didn’t do
— it couldn’t be interrupted once it started talking, ran on credits users
couldn’t replace with their own key, stumbled outside English, broke on a second
monitor, and dropped points whose coordinates came back slightly negative. Tiko started
from that list, with one constraint of its own: nothing paid.
What happens when you ask
- hold
- A listen-only keyboard tap notices the shortcut in any app. A third key cancels, so
⌃⌥→ still switches Spaces instead of starting a recording.
- listen
- The microphone feeds Apple’s on-device recogniser in Indian English, picked for
Hinglish, and the words appear beside the cursor as they’re recognised.
- look
- On release every screen is captured with Tiko’s own windows left out, plus a
full-detail close-up around the cursor — in parallel with finishing the transcript.
- ask
- Gemini gets the question, the last ten exchanges and the images. Model names come from
the key itself, fastest first, and a rate-limited model falls through to the next. A
stronger flash or pro model can be chosen in Settings, with the fast ones behind it. The key
itself sits in the Mac’s keychain, never in a file.
- point
- The reply ends in a tag on a 0–1000 grid, mapped through display points into
the global coordinate space every screen shares. If the element is labelled, Apple’s
text recognition finds that label on a full-detail capture; otherwise a 400-point crop
around the guess is checked again. Then the buddy flies there on a Bézier arc.
- speak
- The system voice reads the reply.
Esc stops the speech, the request and
the pointing at once.
- guide
- Tasks that take several clicks come back one step at a time, each worked out from a
fresh screenshot — the next menu item isn’t on screen until the last menu opens.
Measuring the pointing
Six screens drawn at Retina resolution with macOS’s own font sizes — System
Settings, a text editor’s toolbar, a login page, Finder with a menu open, a save
dialog and a spreadsheet — and twenty-nine questions phrased the way people say
them, each asked twice. A hit is a point on the element or within six points of its edge.
Each run scores the single guess on the full screenshot and the guess after the close-up
check; from run 4, also Tiko’s path of reading the label first.
- run 1
- 29 of 58 single guess · 43 of 58 after the check. Every “no point”
was the comma bug.
- run 2
- Parser fixed: 32 · 49. The check turned seventeen misses into hits and broke none.
- run 3
- Close-up labelled: 47 · 48. The first guess caught up; System Settings rose from
5 to 7 of 10 while the spreadsheet fell from 10 to 6 of 12, where questions about totals
and charts now point at the formula bar or the data.
- run 4
- Label read on screen first: single guess 46 · check on every question 45 ·
Tiko 48, at 1.53 requests a question. The label was found 27 times and hit all 27;
System Settings reached 10 of 10. The spreadsheet stayed at 3 of 12 — there the
model names the wrong element, which no pointing step can repair.
The single guess is Clicky’s method — one guess on the whole screenshot —
run with the same Gemini model and Tiko’s prompt, not Clicky itself, which uses
Claude. The screens are tidier than a real desktop, so the numbers compare methods;
they don’t promise accuracy on your Mac.
Not yet verified
Pointing and the Gemini client have been run against the live API, the flight path and
the settings checked offline, speech rendered into a buffer rather than the speakers, and a
typed question sent through the running app to an answer beside the cursor — but
nobody has yet held the shortcut and spoken to it, and it hasn’t met a second display.
Every build is signed with the same certificate, so macOS should keep its permissions across
updates; that hasn’t been confirmed yet. The release is built for Intel too, but has
only run on Apple silicon, and the switch that opens Tiko at login hasn’t been turned on
yet. New releases are offered in the panel; installing one is still a download and a drag. Without a paid Developer ID, the first launch still needs one
extra click.
Built with less
6,753 lines of Swift in five targets — the app, a core with no interface that runs
from the command line, the benchmark, the screens it draws, and 105 checks — built
with the Command Line Tools alone, no Xcode, over sixteen pull requests.