The Difficulties of Building AI Tools

[ben]

we do these things not because they are easy but because they are hard - JFK

As tech expands exponentially (as it has for half a century now), we find ourselves in a place where we can scale now scale intelligence.

New problems arise as old ones either die or become more prevalent than ever.

Recently, we’ve built out an assessment tool that produces structured feedback and next steps based on a suite of heuristics, rules, and data. It leverages several AI models with complex system prompts that all have some sort of reliance on each other.

Here are a five of the core problems that arose:

1. Prompt Collaboration
2. Unknown and Misunderstood Levers
3. Training Data (and how to do it)
4. Slow Feedback Cycles
5. Measuring Outputs

Here’s a breakdown of each of the problem spaces with a description and requirements to solve the problem:

As we dove deeper into the systems, we’ve developed our own AI stack to solve some of these problems:

• PromptLayer (product)
  • Collaboration: Checks off versioning, labeling, and templating, streamlining collaboration.
  • Feedback Cycle Time: Increases feedback cycle time by being able to pull prompts into production on-the-fly.
• The Five Levers (framework)
  • Defining Levers: prompt & system design, UX, fine-tuning & data-engineering, and model section & architecture.
• Helio Surveys and Benchmark Testing
  • Measuring: We’ve created a few internal testing suites to track our progression over time.

Here are a few resources that also helped our progression along the way:

• Designing AI Products and Features: Study Guide - NN/G
• Most People are Building AI Products Wrong - Here's How to do it Right
• 20 ways to build AI/ML products
• Designing AI Products and Features: Study Guide - NN/G
• How to Create AI Product: Complete Guide From Linkup Studio’s CTO

We’re still learning a TON- even have some other goodies that we’re working on (don’t tell anyone I said anything ).

Curious- has anyone else seen any progress or frameworks to help improve AI products? Also, are there any problem spaces that you’ve seen that I’m missing here?

[ben]

When it comes to training models vs prompting them, I found this insightful breakdown.

Sometimes it’s not just all about “just prompt it bro”.

[MoData]

Which of these does “telling the AI to explicitly do something, and it still ignores you” fall under?

[ben]

Prompting!

Have you ever told someone to not do something in a chunk of tasks they should do and they still do it? (I bet you all the time).

Sometimes, you have to say it more than once to achieve the desired result, but sometimes that is still at the cost of them forgoing something else.

Something that I’ve found super effective is setting constructive rulesets with a title based md format:

## Core objective

Do something...

## Context

Definitions and words and bla...

## **Ruleset**

- DONT DO THIS THING
- Do this thing
- Do this other thing

[nathaliesmith]

@ben

Have you ever told someone to not do something in a chunk of tasks they should do and they still do it? (I bet you all the time).

Sometimes, you have to say it more than once to achieve the desired result, but sometimes that is still at the cost of them forgoing something else.

haha, gotta love free will. That’s what makes humans special

[ben]

Built a little LinkedIn post around this and created a new visual!

The hardest problems in AI have nothing to do with models. Over the past...

[MoData]

These pieces make sense. Can you drop the link to the LI post?

[ben]

My bad, dropped the link! Will drop it here as well

The hardest problems in AI have nothing to do with models. Over the past...