One of the more important parts of my job here at ZURB is consuming and understanding everything that’s happening across services, marketing, and products.
I’m here to try and automate all of it.
Time is a key benefactor to providing immense amounts value. We’ve learned this with Helio, and how much power it provides to fulfill and synthesize tests in a single day vs having to wait for weeks for a research team to fill their quota.
We’re frequently learning about how these new AI tools fit into our ecosystem, and have so for almost a year now (maybe longer, time feels like it’s flying faster than ever).
We’re Defining Processes
To automate anything, we need to be able to define the processes. If we didn’t know what was happening across the org, how could we even begin to automate?
We started with understanding the work: What happens during a project cycle? In our case:
- Meetings.
- Consuming analytics across Google Analytics, PostHog, BeeHiiv, etc.
- Running tests to collect data.
- Creating sketches, designs, mockups.
- Conversations between teammates about the work.
- Synthesizing test data to produce learnings.
These are some of the core parts to our service that makes the ZURB machine provide incredible value, a lot of it backed by extreme amounts of design and business expertise that’s been compounded by technology over the last 20 years.
We Are Frequently Experimenting with AI Tools to Find Leverage
Many of these steps in a project cycle produce something valuable. If they didn’t, shouldn’t they be killed off?
- Meetings provide transcripts, typically a summary of what happened in the work, and what decisions were made.
- Analytical tools give insight into what’s currently happening within the outer layer of a business. The insights here can be invaluable, if you can find them.
- User tests produce data that allows us to benchmark, learn, and make decisions with confidence. Finding those insights can take an exorbitant amount of time.
- Sketches/designs/mockups visually represent decisions or ideas that were realized from previous information mixed in with design intuition about the world and how users perceive it.
AI can:
Ingest surmountable amounts of text to summarize what happened in a short period of time (less than a minute). You can ask it questions about the call, or even create skills (like our glare-design-review skill) that parse data and output it in a specific way. This means that meetings, analytical tools (if API data is provided), and even user tests can all be ingested to produce contextual summaries about what happened.
Problems We Keep Facing
AI only takes you so far.
One problem that we’ve frequently ran into is the quality of the syntheses outputted by AI’s faltering as the context increases without proper structure. Skills, CLIs, an MCP server, and other tools that we’re building are being used to better consume what’s being outputted.
The other problem is context over a project’s life-span. As context shredding happens; where multiple teammates, dozens of AI threads across different toolsets (we’re using both Claude and ChatGTP) are being leveraged, we lose vital context.
I’ve personally noticed how @Bryan can walk through a call, keep the longevity and learnings across months of client work, and spit it back out into today’s learnings – keeping the continuum of the larger picture moving along, and how important having that long-term context in your head is.
We Just Keep Pushing Through
Automation is a tricky thing, even with the advent of AI.
And funnily enough, one of the most important automations that we’ve discovered was a tool that takes multiple Helio tests and compares them to each other, in an easy-to-consume way, which requires ZERO AI to do.
Everything so far has been incremental.
Another skill added to the toolset. Another API endpoint to pull report data out more efficiently. Another CLI that supports building out Helio tests with an agent. Another agent helping us understand our analytics better, and more faster.
Q’s
Would love to hear how other teams are building out their own teams, and perhaps how the thinking is constantly changing over time.
I’d also be curious about what automations or ideas have failed that you thought would have succeeded originally.
