Part 2
This is part 2 of Building Ambi — the story of building a personal AI assistant three times over. Part 1 is here.
Ambi grew the way side projects do. There was never a plan. Every time I wished it could do one more thing, I taught it to. And every time I showed it to someone, they had an idea too — “could it do my calendar?”, “what about the lights?” So it grew and grew, and a few months later it could do an alarming amount.
Underneath, the shape was simple enough. It was a Telegram bot backed by a Gemini agent, and that agent handled most things itself. When something called for deep research or real coding — including writing Ambi’s own features — it invoked a Claude Code CLI session on my Mac and handed the job over. That was the engine. Everything else was layers I kept stacking on top.
And I stacked a lot. Forty-something skills, each a little markdown file teaching it a domain: calendar, email, contacts, finance, GitHub, meetings, research, notes, documents, the filesystem, home automation, habits, goals, even my watch collection. It could listen and talk back — speech in, speech out, a live voice mode I could just chat to. A camera let it notice when I sat down at my desk. It could drive my browser and my smart home. Small local models ran alongside the big cloud ones. There was a memory it kept of our conversations, a score it gave its own answers, reflection passes it ran on itself, and a heads-up display for the things that mattered. And the part I was proudest of: when it hit something it couldn’t do, it could write the feature itself.
Great to demo, impossible to trust.
Read that back and it sounds incredible. Living with it was a different story. The honest truth is I didn’t trust any of it. A change in one corner would break something three rooms away. Features I’d half-finished sat next to features I’d genuinely forgotten I’d built. Every ability I added was also a new way for the thing to fall over, and I was adding abilities far faster than I was adding any reason to believe them. The experiment had quietly become the thing I relied on, and I was still treating it like a scratchpad.
This is the part that took me too long to admit: it always demoed beautifully. You show someone the voice mode, the calendar summary, the lights going off on command, and it’s pure magic. The cost stays invisible right up until you try to rely on it. Then you notice the magic only lands eight times in ten, the other two are unpredictable, and you start quietly checking everything it does. An assistant you have to double-check isn’t really an assistant.
And it isn’t a beginner’s mistake. It’s the default direction of any capable agent in the hands of someone who keeps saying yes to “could it also.” Capability ran miles ahead of trust. I’d built something I couldn’t reason about anymore, and I was proud of the feature list right up until I noticed I didn’t believe a word it told me.
Every feature you bolt on is also a failure you now own. I learned that the slow, expensive way: a long, dazzling list of things it could do, and not one of them I’d stake my morning on.
Next up: the feature I was proudest of and chased the longest — an agent that writes its own features. That one deserves a post of its own.
Leave a Reply