On Research Design: Coffee Loyalty

Just about two weeks ago, the Wall Street Journal published an article by Julie Jargon (is that a cool reporter's name, or what?) discussing the results of coffee drinker loyalty research from CustomersDNA.

I kept the article because of the counter intuitive results presented in the piece, i.e., McDonalds had the highest scores on customer loyalty among the three major US chains- Starbucks, Dunkin' Donuts and the golden arches.

With my background in marketing research stretching back to my undergraduate studies in marketing, I was perplexed that an independent firm which, apparently, conducted objective research, would reach such surprising conclusions.

What would have to be true for that conclusion to make sense?

After all, nearly everyone I know who frequents Starbucks swears by it. Same for Dunkin' Donuts. My own experience at McDonalds, except for outlets located at major highway rest stops, is service so poor that their espresso-based coffees are simply not memorable.

That's when I reread the article for the second time and realized what was wrong.

CustomersDNA only surveyed patrons about coffee. Any and all coffee.

Well, for Christ' sake, McDonalds has been selling coffee for longer than Starbucks has been in business. That's not the point.

In a paragraph located near the end of her story, Ms. Jargon discussed pricing for basic, brewed coffee at the three chains, repeating the contentions of the researchers that McD's lower prices were the prime cause of their superior loyalty measures among customers.

Having realized this flaw in the study, the whole piece became of basically no importance to me. Nor, I would suspect, anyone else actually interested in how the three chains stack up competitively in espresso and non-coffee beverages.

What I can't figure out is why a research firm wouldn't understand this point and design their surveys and methodologies accordingly. After all, McDonalds entered the espresso-based coffee segment recently because of its higher margins and price points. Same with DD.

And, if you've been to Starbucks or Dunkin' Donuts very often, you would, like me, probably see very few basic, brewed coffee orders. Who cares about that segment? Sure, Starbucks introduced Via, but that's primarily for the home market.

It goes back to something my mentor at Penn's business school's marketing department, Jerry Wind, taught me so long ago. You should be able to specify your detailed research plan, down to the output tables and hypotheses tested, before going to the field with your research. But if you have failed to design the instrument and sampling plan correctly, you're already screwed.

In this case, hard as it is for me to believe, apparently CustomersDNA didn't bother to distinguish between espresso and brewed coffees.

Or did they, but one has to pay for those results?

Too bad if it's the latter, because as Jargon's article reads, and this isn't her fault, the results are, well, boring enough to make you need a cup of coffee to stay awake.