Today I wanted to talk about something that comes up frequently when I talk to other people on AI-related topics. Something that can have a huge impact in the final performance of an AI solution depending on how the teams address that.

I am talking about how AI Engineering or Data Science teams should interact with the teams that have the domain knowledge for a specific use case being tackled. For example:

  • the sales team when your company wants an AI solution to improve sales staff productivity, or
  • the logistics team when your company needs a smarter material or cost allocation.

Why is this important? From my experience, it is not enough to have one very knowledgeable AI Engineer another very knowledgeable domain expert.

Clear Analogy

Think of it like this. Tom wants to build an extendable table for his living room. He has a small set of power tools, a local hardware shop with limited materials, and he calls his professional carpenter buddy Jack, who lives some driving hours away.

They go back and forth on the call: what Tom needs, what materials are available, how the design needs to adapt. They come up with a solution. But there was probably a material or type of hardware Tom would never dream could help and was not common enough for Jack to propose during the lengthy and tiring conversation. The table could also fit Tom's living room even better with a couple of alterations Jack would catch if he had seen the room himself.

All of this would change if Tom and Jack were the same person. Or if Tom could easily invite Jack to his home and go to the shop with him. Tom had all of the requirements knowledge and Jack all the woodworking knowledge, but that was not enough.

Same for the domain knowledge of a use case and the needed AI knowledge. Some tools, some materials, or some requirements/pain points only come when all is in the same mind. The Venn diagram of what each side knows should have the largest intersection possible.

How I came to this

This became really evident to me in my time as an AI Product Owner at Siemens. I worked in the Analytics Lab, a team from the central IT unit that focused mostly on delivering AI solutions to the business-focused units of Siemens that did not have a dedicated team for that.

It was our job to reach out to those units to find use cases where applying AI would solve real pain points or give real benefits. Since credits flowed from department to department depending on development time dedicated to projects, there was no incentive to make something just because it was cool, or just blindly try it out. We needed to convince other departments our solution was good, and we could leave the project stakeholders frustrated if we failed to provide value with the final outcome.

To close the gap I eventually hosted some informational AI workshops on time-series forecasting for business unit members. I was also aware of other more interactive workshops done by the Analytics Lab where my colleagues asked for people to fill in pain points, data sources and other relevant information so that they could talk about solutions.

This effort of the team I was in got stuck in my head. It was really needed. Either our business unit colleagues needed to know the tools available to parse the data or we needed to know what solvable pain points they had. Probably not the ones they mostly fill in. We needed to close the gap between Tom and Jack.

I talked to some colleagues that worked for business unit branches with dedicated Data Scientists, and the solutions they had were things I would never have thought to propose. Those Data Scientists had insight into the needs, caveats, and possible gains of the unit. This was possible for the technical team to acquire because their work was much more focused. They could learn the domain knowledge needed to arrive at that.

Natural language use cases

And for the natural language era of AI there are some good examples. Our team at Analytics Lab made a knowledge Q&A chatbot with RAG. Early drafts of the solution offered erroneous responses which were caused by the chunks received from the embedding search.

For example, in an ownership-focused company like Siemens there are common Owner titles for different services, projects and products. Some of these have similar names. It is normal then, that when you ask who the owner of X is, you could retrieve the owner of Y as part of the context if that service/product/project had matching keywords. This was spotted quickly, because we were all from Siemens and knew this.

In fact, there were a lot of acronyms and terms that only made sense within Siemens. So it wasn't that farfetched of an idea to train a dedicated embedding model. Or simpler, to do some string replacement on the chunks that contained these terms. Replacing the Siemens-specific acronyms and terms with synonym text that ensured similar content had lower vector distances. People in Siemens would catch this much sooner than if an external team had made the solution.

What I learned

As an AI Engineer it is very important we get acquainted with the domain we are applying AI to. The more we know, the better. Having a domain expert you can blindly rely on for requirements is not as productive as someone having knowledge, even if limited, of both.

This I did when I later joined Mindmymind. I knew I needed to know the behavioral theory we applied at a very deep level. Second only to the expert team itself. Otherwise we would have missed important ideas that gave clearer next-step directions.

It is also important to incentivize your teams to do this and, when hiring, to probe how much the future team member is curious and willing to understand it too. That curiosity can spark fruitful ideas on how to parse the data, what solutions to use, among others.

Can be the difference between failure and success.