Zero Guesswork

By Jack Elliott, Senior Analytics Engineer, Beam Benefits

Conversation started. Question asked. Answer received.

Except there’s one fairly important problem: can you trust the answer?

AI is great at dealing with ambiguity and filling in the gaps, or at least giving the impression that it's great at it. So how do we counteract this and make sure the same question asked by different people gets the same answer every time? 

We started tackling this problem in March 2026, and in August we launched conversational analytics through Claude chat - more than 70 governed metrics, sliceable by over 150 dimensions, directly from our data warehouse. A way for people across the business to ask questions of our data in plain English and get useful answers back.

At Belfast Tech Week, my colleague Antone Bagnall and I had the opportunity to present what we built, as well as the how and why. It was also a chance to share more of the technical work happening at Beam in Belfast. Our office here opened in December 2025, and the Northern Ireland team has since grown to 19 people.

The conversational part of this project was relatively straightforward. Connecting a large language model to a data warehouse isn’t novel anymore. The harder questions are what happens around that connection. Where does the definition of a business metric live? What is the model allowed to query? How does it behave when the answer isn’t available? And how do you test the system well enough that people can actually rely on what comes back?

Rather than starting with “How do we give Claude access to more data?”, we started with a different question:

How do we give it access to the answers we’re prepared to stand behind?

The hard part wasn’t the conversation. It was the governance.

We already had the ingredients of a modern data stack: a transformed data warehouse with organised data marts, approved content in Looker™ and analysts who understand the business logic behind the metrics people use every day.

The challenge was consistency. Metric definitions can end up living in multiple tools, and once a Large Language Model (LLM) enters that environment there is another risk to manage. AI models are extremely good at producing plausible answers, including when they don’t have enough information to produce the right one.

The problem we outlined at Belfast Tech Week therefore had three connected parts: metrics being defined separately across tools, the need for stronger governance and the risk of hallucination from an unchecked LLM.

Before making Beam’s data conversational, we first needed to make the meaning behind it more explicit.

Moving the meaning upstream

Some of my colleagues attended dbt Coalesce™ 2025, when dbt Labs™ announced MetricFlow. We already used dbt to build, test, document and version-control the transformations that build our data warehouse every day. So naturally, dbt MetricFlow became an ideal platform for building our semantic layer upon.

The semantic layer allows us to define metrics centrally rather than asking each downstream tool to interpret the same business concept independently. A metric can carry not only its calculation, but also the filters, exclusions and caveats that explain how it should be interpreted, along with custom metadata, for example, whether it has been approved for AI or conversational analytics.

It's also important to call out that we weren’t starting from scratch. Business knowledge already modelled in Looker over the years gave us a head start - we translated select LookML models into the MetricFlow configuration, forming the base of the semantic layer.

In the presentation, we used a deliberately simple example to show what the MetricFlow configuration looks like:

The rules for interpreting a metric travel with the metric rather than being recreated every time someone asks a question.

Giving Claude tools, not the keys to the warehouse

Once the semantic layer existed, we needed a controlled way for Claude to interact with it.

dbt Labs offers a MetricFlow MCP server for dbt Cloud™ users; however, we run dbt Core™, so we built a custom metricflow-mcp server that exposes five specific tools. 

metricflow-mcp

Connects Claude to the semantic layer through 5 tools:

• list_metrics → Discover what is queryable
• get_dimensions → List available dimensions
• query_metrics → Get data, appends caveats
• get_metric_details → Metric definitions
• log_query → Log user information, prompt, results

What happens when the answer isn’t there?

A second custom server, looker-mcp, lets Claude search approved Looker dashboards and allowlisted looker agents. 

looker-mcp

Lets Claude search approved Looker content through 2 tools:

• get_dashboards → Find approved dashboards
• get_agents → List allowlisted Looker CA agents

If that isn’t enough, we also hooked up the Atlassian® MCP connector so a question that cannot be resolved can become a Jira® issue that gets added straight to the data teams backlog as soon as Claude receives the users go-ahead. 

That final path has turned out to be useful in its own right. By the time something reaches our backlog, the person asking has already spent time articulating the question and exploring what they need. The request can arrive with considerably more context than a message that simply says, “Can you pull this number for me?”.

The infrastructure around all of these tools is intentionally lightweight. The MCP services (hosted on Google Cloud™ Run) are serverless, so there is very little cost when they are idle, while BigQuery™ handles the compute. Every request carries the individual user’s Google™ authentication, and metric traffic is restricted to our allowlisted gateway.

One of the most important architectural choices, though, is what Claude is not allowed to do.

We don’t let it execute arbitrary SQL against the warehouse. If an approved metric exists, Claude can query it. If one doesn’t, it has to take another approved route rather than constructing its own query and guessing.

That constraint is deliberate. The goal isn’t to maximize how often the system produces an answer. It’s to control where those answers come from.

Then we tried very hard to break it

Before launch, we spent two weeks testing, including two full-day hackathons with the entire data team and hundreds of conversational analytics sessions.

We tested the accuracy of results when governed metrics were available, the handoff to Looker and the quality of Jira tickets created through the conversational flow. We narrowed the issues down to the ones that mattered most, fixed them and tested again.

That testing was important because we weren’t trying to build another demo of an LLM connected to a warehouse. We were putting something in front of people across Beam and asking them to use it as part of their work.

Seven weeks later

It’s still early, but we’re beginning to see the behavior we hoped for.

Direct analytics requests to the data team have fallen, while use of approved Looker assets has increased. Questions that do become tickets are arriving better defined and backed by an example the user has already explored with Claude. We’ve also seen positive feedback and use cases emerge across the business, particularly among senior leaders.

For the data team, the point isn’t to replace analysts with AI. It’s to make more of the repeatable questions genuinely self-service, so analysts can spend more of their time on the work that requires their expertise.

A collection of analytics engineering problems

Getting this far has meant working across data modeling, metric governance, serverless infrastructure, authentication, permissions, tool design and routing, then making sure all of those pieces behave properly together. That breadth is a big part of what made the project interesting to work on.

Seven weeks of real usage is already giving us a better sense of which metrics people want to explore, where the semantic layer needs to expand and where the experience can improve. The next phase is less about proving that the approach works and more about learning from how people actually use it.

For a Beam team that is still establishing itself in Belfast, it was great to take one piece of that work out into the local tech community so soon after opening the office.

Thanks to MCS Group and Belfast Tech Week for having us, and to everyone who joined the session. We’re looking forward to seeing where the work goes next.