Getting Started With Your Data Program

Many of our initial meetings with clients focus on understanding how to get started with data and AI. Our philosophy at Analytics Strategies is that in most cases it makes sense to begin by understanding the existing data and processes both for how the data was generated and how it has been used. By analyzing the data we often discover patterns that are easy to act on to quickly drive value.

However, we are finding more of our clients ask tools like ChatGPT how they should tackle these challenges. In this post, we will discuss the guide from ChatGPT and how our recommendations align or diverge. The ChatGPT output is available here.

1. Identify Business Needs and Goals

The first step suggested by ChatGPT is one we wholeheartedly agree with. However, the recommended "common applications" are somewhat lacking. For example, customer service automation is an opportunity for LLMs specifically, but is only useful if you are trying to cut costs on customer service. It is much less clear how data analysis, predictive maintenance, and personalized marketing are easy to address with "AI." If my question to ChatGPT is really focused on LLMs, the opportunities to drive quantitative business benefits are still somewhat limited. If my question does include data analysis and predictive algorithms, those problems are usually better solved by starting with a simplified "toy" model to assess the opportunity versus the effort. At this point it also is premature to set measurable goals given the fact that AI projects invariably have a research component.

2. Educate and Build Awareness

The second step suggested by ChatGPT seems entirely out of place. Change management is hard enough when we know exactly what we want to change and why. It is unclear how taking a bunch of people and having them attend a bunch of workshops, seminars, or online courses is going to help the company. Having a couple people lead and share what they learn that is specific to your business could be useful. But why not start with just yourself as a leader and then identify opportunities for others?

3. Assemble a Cross-Functional Team

Having a variety of expertise is very helpful for any project. If we were going to recommend education and building awareness above, it would certainly make sense to start by choosing the team first. Separate from that concern, creating a dedicated AI team is not what we usually recommend except to very large organizations that already have a strong data foundation. Most companies can benefit from data and analytics much more easily by just giving people access to the information they need to do their jobs. Focusing exclusively on AI from the start often just introduces risk and in fact can reduce the benefit as it limits which opportunities the team can pursue. We have seen clients who clearly needed basic reporting try to make AI a pillar of the project so they could justify using the earmarked budget for AI to meet their analytics needs.

4. Choose the Right AI Use Cases

The detailed notes from ChatGPT recommend starting small but with scalability in mind. The idea is to dip your toe into AI without a huge investment, while still having the opportunity to have tons of benefit. We are big fans of the "start small" idea, but we tweak it slightly to focus on ease of development and a limited number of impacted users. If you have one really excited end user who can help the organization through AI, you will learn a lot that you can carry forward on entirely different projects. In short, the right AI use case is important, but we think "right" hinges more on how it will be used than the criteria in the link.

5. Data Collection and Preparation

There will always be a tug of war between the business problem we hope to solve and the data we have available with which to solve it. If you identify a problem you would like to solve, it makes a lot of sense to consider what data you have access to that could be "aimed" at the problem in question. Based on the results, you often have to dig deeper into the data or change the problem you are solving. The emphasis in ChatGPT is ensuring you have lots of high quality data. Our experience is most clients do not have lots of high quality data without substantial effort to enrich and improve their data and processes. This is the single most important reason we generally do not recommend focusing on AI until after you have done other work on your data.

6. Select the Right Tools and Technologies

As noted throughout this post, it is rarely smart to dive straight into sophisticated AI like deep learning and LLMs without doing other work in analytics first. SQL, Business Intelligence tools, and even just Excel are almost always a better way to get started. We believe having a skilled analytics professional at the helm who can help flag data quality issues as well as areas of opportunity for analytics is the most important tool you can possibly have. After that, it is mostly buzzwords (though, we do have some favorite tools we recommend to clients as we learn more about their business and their needs).

7. Develop and Train AI Models

Supposing for a moment that your organization does have good data and a reasonable business case for using LLMs / deep learning / machine learning, training the models is an important step in the process. The reason ChatGPT mentions both data scientists and ML engineers is because a lot of AI applications will need to be implemented in very robust ways to actually be useful. Data scientists are generally more focused on getting something working, while ML engineers handle the complexity of keeping it running in a production system. Some solutions that have great quality simply cannot be deployed. So, if your company has gotten to the point of building some sort of AI model, do keep both the idea folks and the implementors close to help build something usable.

8. Implement and Deploy AI Solutions

The focuse of the ChatGPT notes here are on deplying on premise versus in the cloud and integration with other systems. If you hired the right folks in the step prior, you should be all set on the integration piece. If you are somehow considering an on-premise solution for production in 2024, strongly consider making your next project moving to the cloud.

9. Monitor and Evaluate Performance

Before you deploy an AI solution, we believe you should know what benefits you expect to achieve through that implementation. Typically after deployment, the focus should be on how you know if your model needs to be re-trained as well as warnings if it might need to be replaced entirely. Depending on the use case in question, continuous improvement may be worth the effort, or not. We recommend making those judgements on a case-by-case basis.

10. Ensure Ethical AI Practices

In our work, this is not an afterthought, but one of the go/no-go criteria when selecting the business needs. If your organization is considering implementing a non-interpretable model, we believe it is not just the right thing to do, but can make good business sense to think about how bias may lead to worse results through AI. In brief, AI algorithms are great at "cheating" and so if they see a correlation that a marginalized group has worse outcomes, it is likely to copy exactly those patterns.

11. Scale AI Initiatives

We agree that after some initial success, there is a lot of opportunity for AI to really drive value for your company. At Analytics Strategies we are big believers in the power of data and analytics to help your organization grow. As noted throughout this post, the right application, the right tools, the right team, can all help you achieve real benefits for your company.

12. Foster a Culture of Innovation

Our favorite projects are when we get to work closely with our clients as they develop more and more expertise in analytics. We feel "data-driven decision-making" is such a buzz word because people understandably want to make better decisions and using data seems like an obvious way to improve. However, it takes trust in the data, the analysis, and the process for a calculation to change human behaviors. Encouraging experimentation is a great component of a culture of innovation, but access to data is the most common barrier, not skills or culture.

Conclusions

Asking ChatGPT how to be successful with AI can give you some interesting talking points as an analytics consulting company. Following exactly what it recommends as a company looking to apply AI will probably lead to some significant oversights and lost opportunities.