Sometimes AI works better with blinders

When the data needs to be right

If your business uses modern analytics tools, you have probably noticed that your vendors are rushing to add AI to their solutions.

But it’s not clear that the tools are really driving better results.

At Analytics Strategies we have seen that there is usually a large gap between what the data says and how it is used to drive decision making. This gap becomes a chasm when implementing many AI solutions since there are even more questions about the accuracy of the information.

In an ongoing project with a multi-store retail client, we wanted to avoid guiding our users down the wrong path with AI solutions, so we set out to create a system that would more reliably surface accurate information.

We were still interested in capitalizing on the strengths of AI including the flexibility of the systems and the approachability of the interactions for users.

It worked. We got to results that were reliably correct, in 0.2% of the compute time of a naïve pure-AI implementation. Here is how we did it.

Rapid prototyping

In an earlier phase of the project, we had developed and published a set of dashboards designed to provide store managers with the information they needed to make better decisions.

Our challenge now was to create narrative reports that would walk the users through highlights of the published dashboards for a single user’s view. We began piloting AI as a tool for faster prototyping.

A key stakeholder created a sample narrative report. After spending a few hours to set up a semantic model in our Snowflake data warehouse, we were able to simply upload the template and ask the model to fill in the actual data.

Five minutes of computing time later, we had a single representative report which was close to right on most things. There were gaps where the AI system said it didn’t have the correct information, and there were other small differences in the data. But it was promising and gave us SQL queries we could tinker with and improve.

Moving towards production

We wanted these narrative summaries to work as the front door into all the great dashboards we had already built and published. That meant a key success metric was going to be matching the curated comprehensive sales and performance dashboards to the penny as we expect users to dig deep to answer their follow-on questions.

We also had a scale issue: five minutes of computing time per user every day was going to get expensive.

We looked for the current best practices around AI for production deployment. We learned that creating a “dictionary” for the model of today’s data for each user was the place to start. That would allow us to use code for that certified correct index of data, and AI only for the narrative summary of the information.

These systems are still somewhat random, so running the same report on the same data could highlight different information each time. But at least the model would have a much lower risk of running an analysis wrong or pulling the wrong data for a particular user. For this industry and application, highlighting correct data but different information each time is good enough. In finance and healthcare for example, more care would be needed.

Generating results

To implement the design, we used stored procedures and approved queries to create a summary dataset for each user. Then we used AI to generate a narrative summary for each user, one at a time. These summaries were stored in a file for later review but could also be emailed or embedded in a BI dashboard.

It paid off. Instead of taking five minutes per user when we just asked AI to generate the report itself, we were able to run over 100 users’ worth of reports in about a minute. The data also matched the dashboards exactly, instead of only being close to right.

Next steps

We still have more work to do before giving users these narrative summaries and links to the underlying data. But the huge technical win means that the cost of implementing an AI solution will be relatively cheap, while also being more accurate.

We also expect to deliver concrete benefits by making it easier for users to understand their data.