There are a million AI experts out there. Can they all be right?

If you want to find an expert in AI, you don’t have to look far to find a million ‘experts’ offering their skills, training courses, webinars, consultancy…you name it. Everyone is apt to forget that it is in its infancy. There will only be four candles on ChatGPT’s birthday cake in November this year.

I was asked at a conference recently, “What’s your biggest learning about AI?” “The potential” was my first, slightly glib answer, and then I reflected a little more and said, “Learning that it’s OK to be wrong and being prepared to rethink solutions”. In an age where politicians and other leaders are accused of being weak or incompetent for making ‘U-turns’, AI is different from the past.

In the past, most of us have accumulated knowledge, skills and best practices over years and (mostly) got better and better at our jobs. AI is different. Why? Well, AI is still evolving and becoming an expert user in four years is a tall order.

So, how does a company selling MRDCL, a heavy-duty number-crunching tool, incorporate AI effectively? Our first experiments were to take results in crosstabs and turn them into charts and reports, as quickly as possible. Yes, this would be a real benefit, potentially saving hours.

Success was quick, it often is with AI, but the deeper question was how successful. I’ve long had a theory that it is important to differentiate between things where success means OK or good enough, and success means excellence. If you’re travelling on the holiday of a lifetime, success is excellence, whereas if I order a new set of screwdrivers from Amazon, success is good enough.

What we found was that the charts AI generated were OK, the reports it generated were OK, but not exceptional. And when we looked at the wider market, there were a hundred and one other companies at conference trade stands doing the same thing, with varying degrees of success. What’s more, I realised we were going into a market where we are not experts, never mind expertise in AI, we are not experts in charting or writing research reports with the best insights.

While there is undoubtedly a market for ‘good enough’ market research, our expertise lies nearer the ‘excellent’ end of the market. Specifically, when MRDCL, as a software system, makes a difference. For example:

• Complex analysis
• Complex data journeys
• Tracking studies
• Large surveys
• Repetitive requirements
• Multiple reports/outputs

A big rethink was a top priority. We realised our expertise was in survey processing, and survey/data management was the thinking part. How could AI help companies who were serious about these things? If we could utilise AI to make MRDCL more accessible, with built-in intelligence to handle the time-consuming tasks, or tasks that took time to learn, we would have something that was not just ‘good enough’ but ‘excellent’.

A scripting tool like MRDCL has enormous power, commands, and methodologies that a user can take advantage of. Our task was to teach an AI agent the basics of MRDCL and, rather like giving someone a crash course, teach it best practices. This challenge took some time and a lot of groundwork, where thoroughness always beat trying to take shortcuts. Avoiding shortcuts when working with AI is a temptation you have to watch for constantly.

As using the agent can involve describing what you want or uploading texts, the agent needed to question anything ambiguous, unclear, or potentially flawed. Again, building in these guardrails is important, as AI, like an unsupervised junior, can produce what you don’t want.

The results were stunning. We have developed an AI agent that creates variables, texts and table specifications using best practices. What’s more, when we asked the agent to find some deliberately made errors in our files, it found them all and corrected them.

As an experiment, we fed a PDF questionnaire into the agent and asked it to generate an MRDCL script; it worked perfectly, albeit on a fairly standard 40-question questionnaire.

Have we finished yet? No, we’ve got more testing to do, and we’re aiming higher to handle more complex circumstances, where the agent needs to inspect questionnaires and data, determine what is needed, and ask questions as necessary.

We’ve also not finished, as this is Phase 1 of 2. We are close to completing Phase 1, which has focused mostly on generating variables and handling texts. Phase 2 will be taking those variables and producing basic, complex, repetitive analysis, or, indeed, any sort of analysis using an agent. We’re already thinking about Phase 3, but that’s for another day.

So, are we AI experts now? Absolutely not. Perhaps that is the biggest thing we have learned. Four years into the ChatGPT era, being certain about where AI is heading may be less valuable than being prepared to experiment, discover that something isn’t quite right and change direction. We started by asking AI to produce charts and reports. It worked, but we realised that wasn’t where we could add the most value.

Perhaps the smarter approach to AI is not to follow the technology, but to follow your expertise. So, we changed direction and applied AI to an area we genuinely understand: the difficult mechanics of survey processing, analysis and data management. The results have exceeded my expectations. Will we get everything right from here? Of course not. We’ll almost certainly change our minds about some things again. Perhaps, for now, that’s what being an AI expert actually looks like.