The six step framework: Assess

In the previous two chapters I discussed how to Get Ready for AI and how to Ideate. Now that you have a bunch of ideas, it’s time to assess them. In the next chapter we’ll rank the ideas so it’s easier to decide what you should implement first.

I’m going to describe three main metrics by which you can assess AI-ideas or use cases:

• Business value
• Feasibility
• Maturity

I’ll give a framework to put a numeric value next to each metric for each use case but you don’t need to take those too literally. The important thing is that you apply the scores consistently between all use cases that you’re trying to compare.

This and the next chapter are accompanied by a tool that helps you with the analysis, assessment and comparison of your use cases. (Which you can access after subscribing to our newsletter.)

Analysis

Start by further analyzing and writing down a few things about your use case. Do this in a Word document or use our tool (linked above). If you’ve followed our AI workshop template (see How to get started with AI: Ideate) than you’ll already have most (if not all) of this information.

It’s best to start by giving your use case a name. It doesn't need to be too fancy, we’re not marketing to your customers yet. Go with something descriptive but not too long. The most important thing is that you know what it’s about. Also write up a short description of your use case, as a reminder for yourself or anyone who’s going through these cases at a later stage. Focus on describing the problem you’re trying to solve and how you intend to solve it.

Also write down who will benefit from this solution. This helps you with assessing the impact (or business value) of your solution. E.g. if your solution saves all your 200 employees 15 minutes each day, that’s a huge benefit, offsetting a sizable investment. If only 3 employees benefit, the business value will be a lot less. Next, list the tasks with which this tool will help. This will also help you gauge the business value.

Finally try to describe the data needs of your solution, if possible. This might be more difficult, depending on what you could capture on AI fundamentals during the Get Ready step.

Business value

Business value is scored on a scale from 0 to 5. Here is a description for each score:

1. Negative business value, you stand to loose on this solution
2. No business value or business value has not been established
3. Limited business value
4. Moderate business value
5. High business value
6. Game changer or loss of product / service if not done

Assessing the business value in one go is very hard. Are you taking all aspects into account? Are you applying criteria consistently? These are also the kinds of mistakes that I would make when assessing a bunch of use cases. That’s why I try to make a list of what criteria are important in the current context.

Preferably you’d give a score to each of the business value sub-criteria that you listed but, using that list as a checklist is also a very good method. Here is some inspiration on what you can use as sub-criteria:

• Does this tool improve the overall efficiency of the organization?​
• Is there an impact on the company image?​
• Does it contribute to the ESG goals?​
• Will it attract new customers?
• Does it improve job satisfaction?​
• Is there an improvement in usability?​
• Does it reduce costs?​
• Is there an expected improvement in strategic insights and planning?​
• Does it increase response times?
• Will it provide risk reduction?​
• Does it improve accessibility?
• Will it increase sales (or prevent churn)?

Feasibility

Feasibility is also scored on a scale of 0 to 5:

1. Impossible: technically impossible or legally not permitted
2. Hard: possible in principle but many unknowns remain
3. Challenging: this has not been done before but the initial experiments are positive
4. Doable: it has been done by others but for our case there are still some unknowns
5. Easy: this is easy to accomplish and no major setbacks are expected, though some work is still required
6. Effortless: copy or extend existing solution (off the shelf or in-house)

Feasibility is harder to gauge when you don’t have the necessary technical background. I also consult a data scientist when assessing less obvious solutions on how they would tackle the technical challenges. But there’s more going into feasibility than technical solutions. Take a look at this list of questions you could answer (you can again pick-and-choose what factors are important to you):

• Is the AI model easy or hard to implement?
• Is an off-the-shelf or foundation model available?​
• How hard is the integration into existing systems?​
• Is there sufficient data available?
• Is the necessary data accessible?​
• Are the right people and profiles available?​
• Is automating this task with AI acceptable for the people impacted?​
• Are the stakeholders open to using an AI solution?​
• Is there an impact from the EU AI Act?​
• Are there GDPR / privacy considerations?

There is one important difference compared to business value. Business value can be accumulated when evaluating the different criteria, this means that something that is good for your sales and internal efficiency is better overall, a use case that scores very high on only one of these still brings good business value.

With feasibility it’s the other way around: negative results tend to influence the total disproportionately. And this makes sense, if you think about it. The different feasibility factors act as blockers if there is a low score. Let’s say a solution is easy to make, you have all the necessary data and all your employees and customers are very enthusiastic about it, if you can’t do it because it’s forbidden under the EU AI Act… then there’s no way you will be able to get it working (legally).

So when setting the feasibility score, take the lower values into account and when the score would be zero on one of the criteria, it’s zero overall as well.

Maturity

Maturity is the measure of how ready an idea is to take into production. It isn’t so much a metric to choose whether or not to choose this use case but rather what to do next with it. Here is a list of the different maturity levels:

1. Idea ready
2. Business analysis done
3. Self-experimentation done
4. PoC done
5. Pilot done

Every idea or use case should progress through these levels in order. Once you have an idea, perform a business analysis (try to get an answer to all questions going into the calculation of the business value as above). If you have the business value sorted out, try to experiment with off the shelf tools to get an idea of the feasibility (again, look at the suggestions above for what questions might need answering). Now you have new insights on the business value and feasibility, so don’t hesitate to evaluate the solutions you’re considering at this point.

You know what you want, you have some basic insights into what is needed to get the solution working and it’s still worth going forward, then now is time to start building a proof of concept. This does involve some programming or at least the involvement of some programmers or data scientists. In a proof of concept, some custom code is created usually targeted at getting the AI to perform better or to test any other complex part of the solution.

After the proof of concept (PoC) is built again, evaluate the business value and feasibility and decide whether to move forward, or not. And moving forward means building a pilot. A pilot is a fully functional implementation of your solution (so not just part of it to test feasibility like a PoC). Also important with a pilot is to start integrating with your systems. A pilot doesn’t have to be fully featured or integrated but it should be delivering business value once done.

Finally when the pilot is done you can go into industrialization and iterate on the application to continue improving the features and the business value.

We’ll come back to pilots and iterating in later chapters. In the next chapter, Identify, we’re going to decide what solutions to prioritize.

‍