Over the last several months building AI tools, I’ve realized teams have a handful of distinct “levers” to shape performance — and knowing which lever to pull first is half the art.
After formulating this, I had AI sketch out a simple diagram to help communicate this (kind of funny, honestly. it did it in text!)
Im not sure it matters but we are building a fairly complex translation bot for English to arabic where you input a doc in English and get it back in arabic. One lever that we are using to improve accuracy and minimize token spend is a data pipeline. A smart architecture to orchestrate/loop the right llms and business logic layers is a useful lever.
Definitely matters- sounds very similar to how we’re building out one of our assessment tools to parse websites to output tangible feedback.
I haven’t found training data as useful as I thought it’d be (yet), but that’s also due to not knowing how it’s actually affecting precision in scoring.
Have you taken advantage of evaluations? I feel like we should start to leverage that more, especially as the system becomes more and more complex…
Do you mean like a maker checker? We do but havent found it useful. We were consistently hitting 97% but in reality the output wasnt great. We are now relying on hitl till we can make it the output more reliable. Im really avoiding going the path of another set of fine tuned vectors before the data pipeline hits the business logic layers. Its a pain in the wrong end but we might end up having to do it.
Interesting. Seems mostly about right @ben Like the thinking!
I’d argue that UI/UX layer is fairly easy to change, but the evaluation loop isn’t like traditional user feedback, since the integration of the user ideas is not at the UI/UX layer, but affects all the blocks.
How about something like this, where the X and Y axes are really about how fast you can improve AI.
I reframed this chart around the evaluation loop because every layer, from prompts to models, depends on how feedback is gathered, interpreted, and applied.
It’s not just about iteration speed or system depth anymore. Teams have to think about how well each layer learns from its own outcomes and feeds that learning back into the next cycle. This can be a bit maddening based on our experience.
I say this chart is very accurate. In terms of summarization, I feel like “Training” instead of “Architecture” fits. Also stating “UX” feels more accurate than “Experiences” (even though you might argue they’re the same thing).
What decisions do you foresee someone making by using this framework? On the face of it I agree with Ben’s changes. You can also add model switching or toggle to improve outputs as belonging in Q2.
I could see this as a progression or roadmap. Example:
Initial stages: Focus on the 80%, AKA get to an MVP fast which includes prompt iteration, system design and building the UI/UX Layer around it.
Simplify and Expand: Use MCPs, RAG, & API’s to reduce core complexity (of the prompt itself) and replace hard-coded tasks, while also making context more explicit.
Focus on Accuracy and Repeatability: Train models, synthetically create data, and focus on systems that enhance accuracy and repeatability (thinking that this can be the last 5 - 20% depending on how vital accuracy is to the feature).
Of course you could play with each tool in different orders, it’s very dependent on the need. I’d think that most orgs would probably follow the above structure based on my experience.
