Zen, Betty Crocker, and End-to-End Testing

I don't know if this story is true or not, but I heard it somewhere and it makes a point, so here we go.

A maker of cake mixes was investigating two cake mix designs. One, their traditional recipe, included powdered egg in the mix, and a new one required the end users to add their own eggs. Chemically, the result was about the same, but a competitor's "add your own eggs" mix was starting to gain market share over their existing all-in-one product. They decided to do some market research to find out why.

They conducted a taste test in a statistically valid fashion, with double blinds, etc. Presented with two cakes side by side, tasters on average had no particular preference for one mix or the other; they tasted the same.

This was very nice, but didn't explain the problem: why would the new market entrant be getting so popular, then? They had less advertising, no name brand, and you had to add your own eggs!

They had an idea and decided to run another test: this time, the tasters had to also bake the cake themselves, and they obviously knew which cake was made from which recipe (but were hidden from the brand name, packaging, etc). In this test, the tasters clearly preferred the cake where they had to add their own eggs.

Why?

They had more of an emotional investment in the egg-added cake; it contained "fresh ingredients" that they had to buy and add themselves. It took a tiny bit more work. They didn't feel like they were cheating as much when they made a mix that didn't have all the ingredients built in.

Ironically, we tend to believe that people buy cake mixes to save time and effort (ie. they're lazy); this makes the "add eggs" mix seem inferior. But what the researchers discovered was that people buy cake mixes not just to save time, but because they don't feel competent to add all the ingredients themselves (ie. they're scared). Making the end user do slightly more work makes them feel more productive and gives them more emotional involvement in the end product, without scaring them away with complexity. That's the first lesson.

The second lesson is about unit testing; the cake mix engineers were looking in the product specifications for why it wasn't selling, and constructed a perfect unit test to identify and help correct their assumed problem (if the eggs-included recipe had measured worse, they could have improved the recipe). But the test, while scientifically and statistically valid, included some hidden assumptions that also needed to be tested: that taste affects the desirability of the product; that less end-user effort is an improvement. The proper test to find the problem was closer to an "end-to-end" test: it included much more of the "cake mix consumption" process in the test, not just the part where you eat the cake.

And the third lesson is about the Zen of cake mixes and how capitalism has helped us get closer to it. Years ago, under adult supervision, I remember using cake mixes that didn't require you to add your own eggs; they were just fine. But how many mixes can you find nowadays that don't require eggs?