AI · on-device · September 2026In use

Atlas 1.0

A small language model I fine-tuned so SparrowOS understands plain-English and Hinglish commands, running privately on each person's Mac.

My role: Sole developer: benchmark, training data, fine-tuning, evaluation and the Mac integration

88.6%

correct on a frozen set of 297 held-out commands

~50%

for the best of 34 open models before fine-tuning

33 / 38

standup commands right, up from 12 with the previous model

~330 ms

per command in testing on a Mac, with nothing sent to the cloud

The brief

SparrowOS has 49 actions: create and move tasks, post in chat, log time, pull a brand's analytics, find a file, send mail and more. The team wanted to drive them by voice or a typed sentence, in English or Hinglish, without sending their work to a cloud model.

So the model's job is deliberately narrow. It reads a request and returns which action to take plus the exact words that matter, like the task, the person or the date. Code does everything else: it resolves names, checks permissions, asks for confirmation, executes and keeps an undo.

What I did

Benchmarked 34 open models on calling these exact tools. The best managed about half, and models built specifically for function calling lost decisively, so fine-tuning was required, not optional.
Built a training set of about 3,500 labelled commands against the real tool catalogue, in English and Hinglish, and checked it for overlap with the test set.
Fine-tuned Qwen3.5 2B (4-bit) with LoRA on Apple MLX, then averaged checkpoints from within the run.
Measured it on a frozen 297-case test set that was read once, at the end.
Shipped it inside the SparrowOS Mac app behind Option+Space, with on-device dictation that refuses to fall back to the cloud.

How I did it

Benchmark before building

Every candidate got the same prompts and was scored on choosing the right tool and filling its arguments. Qwen3.5 and Gemma led at roughly 48 to 54 percent. Function-calling specialists (xLAM, Hammer, ToolACE) came last.

Train small, on the machine it runs on

A 2B model in 4-bit fits comfortably beside everything else on a work Mac. LoRA on Apple MLX kept training local; bigger experiments later ran on cloud GPUs. Averaging checkpoints within one run helped. Averaging across separate runs did not, because each run settles somewhere different, so that idea was dropped.

Make the number honest

I found that earlier averaged models had been picked using the same 297 cases they were then scored on, which inflated results by about 4 points. Selection moved to a separate 200-case validation set, and the test set is now read once, at the end. A power check showed 297 cases can only separate differences of about 5 points, so anything smaller is reported as not distinguishable.

Say no to things that looked good

A vote between three models added 1 point that wasn't statistically significant, at 3.5 times the latency. DoRA took 798 ms per command against LoRA's 330 ms. A later experiment that put the whole tool catalogue in the prompt scored 12.1 points lower and was not shipped. One corrupted run, where padding in the prompt leaked the length of the answer, was caught and quarantined. A corpus validator and 36 regression tests, each one a bug that really happened, now guard against repeats.

Ship it where people work

Atlas lives in the Mac app. Hold Option+Space in any app, speak or type, see what it will do, confirm. It is the first model in SparrowOS Intelligence, the family name for models trained in-house.

Atlas on my Mac, recorded on 6 November 2026 and not sped up. I ask what's waiting on me out loud, and it works out the answer in about half a second. The only edits: trimmed, and cropped to this part of the screen.

What it changed

Before and after fine-tuning

Before fine-tuningAtlas 1.0

Show as a table

Atlas: commands understood correctly, 297 held-out cases
	Before fine-tuning	Atlas 1.0
Accuracy	about 50%	88.6%

88.6%

on 297 held-out commands, against about 50% for the best model off the shelf

33 / 38

standup commands right, against 12 for the model that had been shipping

The team can run SparrowOS by voice or a sentence, in the way they actually talk, on their own machines. And the numbers quoted for it hold up: they come from a test set that was never used to make a choice.