AI analysis is only as good as the rules you set

Using AI
1

Came back to Caitlin Sullivan’s piece in Lenny’s Newsletter names four ways AI is messing up your synthesis. Shouldn’t have to create a rule that you have to remember to paste in every single time. That’s the case for Skills (here’s looking at you Glare)… we should consider this thinking in our findings skills.

Lessons from the article worth building into our Skills:

Quote rules. Grab the whole thought…from where it starts to where it ends. Keep the “maybe” and “I think” bits, they show doubt. Keep the emotion. Don’t glue together things said at different points. Tag each quote with who said it and when.

Check the quotes. After the analysis, make sure every quote is real and word-for-word. Flag anything that’s close but not exact, and mark anything you can’t find. This catches the fake quotes before they end up in a deck.

Wondering if Calude is already doing this now?

Give it the right context. Four things, not a rambling brain-dump…

  1. the decision you’re making and why it matters
  2. what a “yes” would need to prove
  3. the product background that changes how a comment reads,
  4. who each person is so their input gets weighed correctly.

Don’t let it chase the obvious. Models grab the theme everyone shares and miss the one comment that’s the real signal. Tell it to surface the tensions and the outliers, not just the majority.

Anyone here already building their analysis rules in like this, how do you bake it into your workflow

1 Like
2

Where this gets interesting, and actually a lot of my AI orchestration work is inspired by the idea of “AI layering”.

Just as Caitlin’s recommending throughout the article, you don’t just input one huge prompt that does everything, you break it down. Start with pulling quotes, then do another prompt to validate them. Running a single prompt to do this is less effective because of LLM’s prediction systems. It’s much more accurate with specified context.

You won’t be like these Startroopers, spraying and praying, hoping that they’ll finally hit a Jedi.

Star Wars Stormtroopers in Hallway

The other part is defining what a good output looks like. Without it, it’s hard both for the LLM to know what’s actually useful for you and your context.

Lastly, layering is awesome because then you’re able to judge the stages, not just the end results. If you end up judging the end results, it’s significantly more difficult to find out what’s going wrong.

Of course, people want FAST. But, I’m finding more and more that when I force myself to slow down, I move faster, especially when orchestrating several agents in parallel.

Clarkson's Ariel Atom Power Test on Top Gear

Great callout here with Caitlin’s post @Bryan! Really liking her thinking.