Match Rates Unmasked: Unraveling the Mystery of Business Traffic Identification!

Navigating Web Traffic De-Anonymization with Demandbase

A core component of Demandbase and our market is the ability to de-anonymize web traffic. One trending evaluation tactic our competitors take part in is saying “We have the highest match rates in the business.” And they recommend a data bakeoff with other vendors to “prove it.”

For transparency purposes, Demandbase participates in account identification data bakeoffs all the time. We win some, we lose some. I’m proud to share that we win more than we lose – but it’s close. In the more than 50 account ID data bakeoffs I’ve been CC’d on in the past 2 to 3 years alone, I’ve come to conclude the following:

There are big problems with this. Let me explain why before offering a quick recommendation.

Problem # 1 – The Definition Problem.

What is the definition of “match rates”?

1. The % of traffic matched to a business (after removing bots)
2. The % of traffic matched to a business (before removing bots)
3. The % of traffic matched to businesses, consumers, and bots?
4. The % of business traffic above an “80% confidence score”
5. The % of business traffic above a “50% confidence score”

If you were to perform a “data bakeoff” between two vendors above, it would not be an apples-to-apples comparison. Even if you deployed the tag on the same pages and compared the match rates using the same timeframe.

Problem # 2 – The Methodology Problem.

Every vendor has a different methodology. At Demandbase, we only identify business traffic that we have determined (via AI-based conclusions) to be the correct account. We are not afraid to publish our global accuracy rate of 92%, meaning we accurately identify the accounts 92% of the time. We follow this methodology because we strongly believe that incorrectly identifying accounts is worse than no identification at all. Incorrect identification will lead to bad decisions and perhaps tunnel vision sales towards the wrong accounts. This methodology of ours also allows the accuracy of our intent and advertising impressions to remain high, but we won’t get into that here.

Other vendors use a customer-facing “confidence score” along with their account identification/graph. This results in higher match rates, but it naturally produces more inaccurate identification rates – “Hey, we share the confidence score, so it’s your fault if you action on it when it’s not 100% confidence”. Using this methodology, these vendors tend to be the loudest in terms of you hearing “we have the highest match rates” or “we identify more accounts compared to our competitors.” For organizations that only care about “match rates,” regardless of correctness or not, this vendor would look appealing to you.

Problem # 3 – The Selective Statistics Problem.

Without sharing the entire methodology and definitions, every vendor can appear to have the highest “match rate” by simply tweaking their “match rate” definition, among other things. They may even share a fancy slide summary, which proves the match rate percentage is correct. Furthermore, vendors that utilize a confidence score can easily move the lever to artificially inflate the match rate percentage whenever they want for the purpose of looking better than a competitor.

Problem # 4 – The Accuracy Test Problem.

Since you now understand the three problems above, you may want to perform an accuracy assessment. The most obvious method to perform this test is to take your first-party data and compare it with the identification data each vendor shares with you. This is generally a pretty sound comparison, but this measures the accuracy of the data, not the “match rates.”

However, this test is not without bias. Suppose two vendors are comparable in terms of accuracy. In that case, the inclination is that the “winner” of the bakeoff between the two vendors is based on who has the highest match rate. After all, an 80% match rate vendor with only 10% inaccuracy is better than the vendor that came in at a 65% match rate with the same 10% inaccuracy, right?

Unfortunately, this doesn’t work. Your first-party data likely has more established companies with a larger digital footprint. Therefore, your data will organically have more accounts in the higher confidence score category and you avoid testing the accuracy of the smaller organizations that each vendor attempted to match, who have a smaller digital footprint, that ultimately don’t matter to your business.

My Recommendation

If at all possible, please do not put a lot of weight in a vendor’s “match rate” when making a purchasing decision. At this point in time, there’s a couple of vendors that have a very comparable offering. There’s an extraordinary amount of other considerations that should be part of the equation for evaluating account intelligence technology. This buying guide might help.

If you must go forward with a comparison test, I highly recommend that the comparison is done in Google Analytics. A single slide presentation or using one vendor’s recommendation on a bakeoff methodology will almost always be biased. It also takes the attention away of what’s more important.