Instinctive versus Expert responses with Generative AI

Generative AI models can give reasonable and convincing responses to a vast array of queries, but there's a marked difference between zeitgeist knowledge that comes during its creation and the results that can be achieved with the proper application of techniques and context.

Instinct

If you were to ask me (or an an AI) how one might improve code quality or how you might reduce duplication, without any prior knowledge about your codebase I’d call out things like definitions of what exactly code quality is, how you could align it with metrics and measures, and give some vague advice about how refactoring code to improve those measures would improve that quality.

Expertise

If you pressed me to give specifics and details I’d need more context, and some time to analyse at the code, form mental models, and perhaps shortcut that path to understanding by aligning it back to some of those previous stereotypes and understandings.

The same is true of Generative AI Models, and is one of the reasons why there’s so much buzz about things like Retrieval Augmented Generation (RAG) and techniques like chain-of-thought and multi-step reasoning.

Instinct is the difference between knowing that a task could look like X and Expertise is knowing that in this specific scenario it looks like this.

I’ve been learning more about Generative AI and plugging it into my tools, local development environments, and automation pipelines recently and from what I’ve learnt, there’s a hierarchy of techniques that you can apply to improve the quality of responses from these tools, which is:

  1. Prompt Engineering and good instructions goes a long way
  2. Context Stuffing and the approach of adding more and more context into your prompt gets you better results (but it can only go so far)
  3. Adding Tools (such as giving a model the ability to fetch additional context on-demand) goes much further than context-stuffing
  4. Retrieval Augmented Generation Attaching a VectorDB store and creating a “RAG system” gives you a step-change in response quality over any of those other techniques

I feel like I am by no means done on my learning Journey here, but my advice to anyone on the same path is: explore RAG stores for yourself, and see how you can take your AI Agents from being models with well crafted prompts to models that are experts in their domain.

I suspect that the next step for me is re-applying all of these techniques to my RAG system - adding decent Prompts, better tools, and hopefully a bit of recursive goal setting / goal evaluation - and hoping that will yield the elusive goal of building an AI Agent that’s at least as good at my job as I am.