The Difficulty of Triangle Testings - A Practical Example

drew's picture

Let's talk the triangle test. You know that simple thing we try and do to prove that changes in brewing do or don't make a difference. We put a lot of stock in the conclusions of that test. What does it really mean?

On the face of it, it's a simple prospect - "Dear Sir/Madam, I present you three glasses. Pick the one that's different." It feels so absurd, so childish, so easy that it's insulting that you'd be asked to choose. 

All of us - ok, almost all of us - have enough belief in our powers of taste, observation, deducation, umm... tactics? - whatever... We all truly think that unless it's something truly miniscule, we've got this. Give me those silly glasses!

And yet... it's really, really, really hard. (Three really's because three glasses)

The other week, Denny and I were in Sydney after attending the wonderful Australian National Homebrewers Conference in Melbourne. Paul Nickodem invited us to come up and give a talk about experimentation.

So we set up at Batch Brewing (you'll be hearing more from them in Episode 80 of the podcast) and we decided we'd show people how to run a triangle test as a "practical" example - aka "stop listening to us talk for a few moments and watch this magic trick", the cornerstone of real entertainment.

The Best Sort of Brewery Visitor

We didn't have an experiment to run, no grand thesis to prove or data to collect - just a simple demonstration.

The Targets

For that purpose, we chose two "similar" Batch Brewing beers for our testers to taste. One was their "Just Beer", a 4.5% plain lager - a simple beer with fruity Aussie hops, made for warm weather and easy drinking. The other - their Das Bohemian Pils - clocks in at 5.0% full of Saaz. Different hops, different strengths, different base malts - this should be an easy test! (And a delicious one as well!)

During our talk, it came time to do the triangle tango. We walked people through the ideal test. We could tell that the audience was questioning - "really? how hard could this be?".

Three brave volunteers raised their hands and they were brave because let's face it, it had to be a setup. Why else would we do it? (Note - all three were experienced homebrewers/drinkers.)

Each taster got a flight of three glasses - sparkling clear glass glasses (see below) - you know, the one's you're not supposed to use for a triangle test to avoid visual influence. It's the best possible scenario for tasters. Except that it was at night, in a noisy brewery with lots of people staring at them. (Ideally, opaque cups, quiet, smell free environment with plenty ofl lighting)

You Can Tell The Difference Between These - Right? (Just Beer on left, Das Pils on right)

They tasted, smelled and furiously thought through the three samples. I've seen less intense game faces at technical interviews. When everyone indicated they had a decision, we called for a show of hands.

The result - and this shouldn't be a surprise given that I've typed this many words - all three tasters picked a sample that was part of the paired beers. They all skipped the "different" beer - in this case, the Pils.

This is really hard stuff!

I watched one taster go through the beers and almost immediately choose the different beer. He then went back through and tasted again. He then talked himself out of his initial choice. 

Takeaway

The takeaways on this little test, because, as we said, it wasn't about data.

  1. Triangle tests are deceptive. They seem an easy thing, but in truth are hard
  2. This is true even when the difference "should" be obvious. (In this case two related, but entirely different styles/recipes)
  3. Any set of triangle tests that reaches the threshold of significance really should standout.
    1. The Flip Side Corollary - a triangle test that fails to reach a significant p-value doesn't really mean "no difference" as much as "not blindingly obvious difference" 
  4. Props to anyone who tastes one of these tests - you're probably going to "fail"- it's the nature of these things!

A Final Word About the Triangle's "Real" Purpose

 

The Man To Blame For T-Tests

That man was William Sealy Gosset and he was a statistician in the employ of Guinness Brewing. He developed the Student's T-Test (had to call himself Student because Guinness didn't want others to discover that they were using math/science to improve their beer) to help determine if statistically small samples, like a tasting panel were indicating something meaningful.

 In the homebrew world, we tend to look at these tests and say "so does this make a difference?", treating it like a curiousity. Maybe we're looking to save time or effort. 

But make no mistake - for a commercial brewery - the purpose of the triangle test is almost always "can we make the beer for less" - whether it's by substituting cheaper ingredients or using more efficient processes. It's all about shaving pennies from each unit sold because that's the nature of business. 

Inherently that's a perfectly sensible proposition. We're trying to save time/effort, they're trying to save money.

But it's important that we don't lose site that incremental changes, like those ideally tested by a triangle test, can ultimately lead you far away. Take a look at the disaster that befell the one time #1 beer in America - Stroh's. They pushed their process improvement too far with ultimate consequences for the company. (Their former glorious LA brewery is now a used car auction lot.)

Regardless.... Triangle Tests are hard and thanks to the fine folks at Batch Brewing for letting us prove it with their beers!

Peace!