The Semantic Layer and Data Modeling Dilemma in AI Analytics

As we’ve expanded AI analytics through Hex and our in-house AirDS, we’ve seen strong demand and a clear uptick in self-serve data usage.

AI is not just helping people answer existing questions faster. It is making teams more exploratory.

Commercial and product teams can investigate customer trends more directly and iteratively, combining quantitative signals such as usage, conversion, and retention with qualitative context from customer conversations, feedback, and support interactions.

Questions that previously required a formal request to the data team are becoming an ongoing conversation with the data.

But it has also exposed a recurring trade-off:

• The more time we invest in the semantic layer and data model, the more accurate and consistent the answers become. But it can also slow us down.
• Do too little modeling, and AI may produce a convincing answer using the wrong metric, grain, or join. Do too much upfront, and every new question becomes a data engineering project.

Our current approach is:

1. Model the core - explicitly define the metrics, entities, and relationships that must remain consistent.
2. Build a curated context layer - capture the lineage, definitions, business knowledge, and analytical guidance the AI needs beyond the core model.
3. Manage that context rigorously - the data team decides what belongs, which sources are authoritative, who owns them, and how they stay current.
4. Run offline evals - we have a Staff Data Scientist dedicated to defining the evaluation framework and testing the system before it scales.
5. Run online evals - the broader data science team reviews real outputs, identifies failure modes, and turns those learnings into better context, models, and evaluation cases.

The data science team is not disappearing. It is becoming the human feedback loop that evaluates the system and continuously improves its context.

Our operating principle is becoming simple:

Model the core. Curate the context. Evaluate continuously.

I’m curious how others are solving this. Where do you draw the boundary between semantic modeling, curated context, and model flexibility?