Measuring What Matters — The Metrics That Actually Tell You If AI Is Working

Every operator we talk to asks the same question within the first five minutes: "What's our automation rate?"

It's the wrong question. Or at least, it's not the only question.

Automation rate tells you how often your AI responded without a human stepping in. It says nothing about whether that response was accurate, helpful, or left the guest feeling good about their stay. A 90% automation rate with a wave of angry guests flooding your reviews is not a win. It's a liability. Meanwhile, an operator running 50% automation with high guest satisfaction scores and glowing reviews is building a business that scales.

The metric obsession is understandable. Automation rate is simple, easy to benchmark, and feels like proof the AI is "working." But it's a vanity metric in isolation. It only means something next to the numbers that tell you how well it's working.

Here's the full dashboard you should actually be watching:

• Automation rate — volume handled without human intervention
• First response time — speed of the AI's first reply
• Resolution time — total time from first message to resolved issue
• Guest sentiment score — how guests feel during and after conversations
• Escalation rate — how often the AI hands off to a human
• Knowledge base hit rate — how often the AI finds a relevant answer
• Review score trend — downstream impact on your public ratings

TL;DR

Automation rate is table stakes. The operators winning with AI track seven metrics: automation rate, first response time, resolution time, guest sentiment score, escalation rate, knowledge base hit rate, and review score trend. Track daily and 7-day rolling averages for each. Quality without volume is slow; volume without quality is dangerous.

The 7 Metrics That Actually Matter

1. Automation Rate

What it is: The percentage of guest messages resolved without any human intervention.

This is the starting point, not the finish line. Automation rate tells you your AI's volume capacity, but nothing about its quality. Track it, but always read it alongside the metrics below. A healthy automation rate for most hospitality operations sits between 60% and 80%. Below 50% suggests your AI isn't trained on enough scenarios. Above 85% warrants a close look at guest sentiment to make sure nothing is slipping through.

2. First Response Time

What it is: How quickly the AI sends the first reply after a guest message comes in.

Guests expect fast. Research from Salesforce consistently shows that response speed is one of the top drivers of customer satisfaction. In hospitality, a guest locked out at 11pm doesn't want a response in two hours. They want one in two minutes. Target under 60 seconds for AI-handled messages.

3. Resolution Time

What it is: The total time from a guest's first message to the issue being fully resolved.

First response time and resolution time are different problems. Your AI might reply instantly but take five exchanges to actually solve the issue. Long resolution times often indicate a knowledge base gap or a conversation flow that needs tightening. A good conversation engineer will look at the transcripts behind the slowest resolutions first. Target: under 5 minutes for standard inquiries.

4. Guest Sentiment Score

What it is: A real-time measure of how guests feel during and after AI-handled conversations.

This is the metric that exposes the automation rate trap. You can have an AI that replies to everything and still leave guests frustrated. Sentiment scoring uses natural language processing to flag negative, neutral, or positive exchanges. If your sentiment score is trending down while your automation rate is trending up, you have a quality problem masquerading as a performance win.

5. Escalation Rate

What it is: The percentage of conversations the AI hands off to a human agent.

A high escalation rate signals that your AI is hitting the edges of its training. A low escalation rate is only good if sentiment is also high. The goal is appropriate escalation: the AI handles what it can confidently handle and passes off what it can't. Target: under 15% for a well-trained system. If you're consistently above 25%, your knowledge base needs work.

6. Knowledge Base Hit Rate

What it is: How often the AI finds a relevant answer in your knowledge base versus returning a generic or failed response.

This is the diagnostic metric. When hit rate drops, everything else suffers: resolution times climb, escalations spike, sentiment dips. A low hit rate is almost always a content gap, meaning your AI wasn't trained on the questions guests are actually asking. Review low-hit-rate queries weekly and use them to fill the gaps. Target: above 85%.

7. Review Score Trend

What it is: The directional movement of your guest review scores over time, correlated with AI adoption.

This is the ultimate downstream signal. If your AI is doing its job, review scores should hold steady or improve as you scale. If they're declining after AI deployment, something upstream is broken. Track this monthly and look for inflection points that align with changes to your AI configuration or knowledge base.

Healthy Ranges at a Glance

Use this as your baseline. These ranges reflect what well-performing hospitality operations typically see after 60 to 90 days of AI deployment.

• Metric: Automation Rate — Healthy Range: 60% – 80% — Warning Sign: Below 50% or above 85% without quality checks
• Metric: First Response Time — Healthy Range: Under 60 seconds — Warning Sign: Over 3 minutes
• Metric: Resolution Time — Healthy Range: Under 5 minutes — Warning Sign: Over 15 minutes
• Metric: Guest Sentiment Score — Healthy Range: 80%+ positive — Warning Sign: Below 70% positive
• Metric: Escalation Rate — Healthy Range: Under 15% — Warning Sign: Above 25%
• Metric: Knowledge Base Hit Rate — Healthy Range: Above 85% — Warning Sign: Below 70%
• Metric: Review Score Trend — Healthy Range: Stable or improving — Warning Sign: Declining after AI deployment

One important note: these ranges are starting points, not ceilings. As your AI matures and your knowledge base deepens, you should expect automation rate and hit rate to climb while escalation and resolution times fall.

How Olala Does It: Daily vs. 7-Day Averages

One of the smarter operational decisions we've seen is Olala's approach to reporting cadence. Rather than checking metrics in isolation, their team tracks both the daily figure and the 7-day rolling average for each metric side by side.

Why does this matter? A single bad day can be noise. A bad day that's pulling your 7-day average down is a signal. Conversely, a single great day can make you complacent if the weekly trend is actually flat.

Daily checks catch fires. Weekly trends reveal the real trajectory.

If you're only looking at one or the other, you're either overreacting to noise or missing slow-moving problems. Build both views into your dashboard from day one.

What Conduit Surfaces for You

Most AI tools for hospitality give you automation rate and call it a day. Conduit's reporting dashboard surfaces all seven of these metrics in one place, with daily and rolling averages built in. You can drill into individual conversation transcripts to see exactly where the AI succeeded, where it struggled, and what knowledge base gaps are costing you resolution time.

Operators on Conduit have used this visibility to cut escalation rates by identifying recurring questions that weren't in their knowledge base. They've caught sentiment dips before those showed up in reviews. And they've scaled their portfolio without scaling their support headcount.

If you want to see what your dashboard would look like with your actual operation's data,book a demo with our team. We'll walk through your current setup, identify where the gaps are, and show you exactly what a healthy AI performance baseline looks like for your property count and guest volume.

Automation rate is a starting point. The operators building durable, scalable guest experiences are the ones tracking everything else too.