I often receive a standard response when I write about cooking myths or food myths here on Culinary Lore. So and so performed an experiment on this which proves you’re wrong (or correct, as the case may be). I don’t often include a comment field in my articles, but some readers are passionate enough about the subject to take the time to email me their thoughts. Responding, however, presents a problem.
First, the reader is assuming I was not aware of the “experiment.” They are usually aware of these experiments because they read about them in articles. If they found such sources, you can be sure I did as well. But, trying to tell someone after the fact that you were aware of a certain source but chose to ignore it is difficult. They can always assume you are lying and “did not do your research.”
Of course, I have failed to uncover valid research in the past so any reader is justified in bringing up such research. However, not all research is created equally. I failed to uncover newer scientific evidence on the question of whether apples keep potatoes from sprouting, leading me, initially, to an incorrect conclusion.
Assuming that the reader does believe that I knew about the experiment, how do I explain the reasons why I chose to not include the sources in my article? I am left pondering the futility, if not the pretentiousness, of explaining the basic scientific method to them. But, that is often what this all boils down to, proper experiments are done by trained scientists, not by food bloggers (sorry, fellow writers).
Let me give you an example. I asserted, in this article about marinades, that a marinade will not really do much to tenderize meat. Let us suppose that a reader, upon finding an article by Mike at Experimental Food Blog (hypothetical blog), contacts me and helpfully informs me that Mike over at EFB experimented with marinades and found that they do indeed make steaks more tender.
Let’s further assume that I already knew about Mike’s article and that I did not consider his experiment or his conclusions valid. Why? Well, it is a far cry from the type of evidence that led me to update the article I mentioned above.
I chose the tenderizing of meat for a reason, and before I begin, I would like to thank Jeff Potter, author of Cooking for Geeks: Real Science, Great Cooks, and Good Food, for reminding me of this perfect example: The Mythbusters, Adam Savage and Jamie Hyneman, performed an experiment to test whether you could tenderize meat with explosives (Season 6, Episode 10). I am a huge, huge fan of Mythbusters and I still haven’t quite gotten over the show ending (bring it back!) You can watch the full episode here on Discovery. Sorry to the new Mythbusters but they lack the magic sauce.
Despite how silly such a notion may seem, the Mythbusters knew something that Mike and many other food experimenters don’t know: Setting up such an experiment and analyzing the results is not a simple matter.
Mike said his steaks were more tender. Why, because he himself cut into one or two and thought to himself “now that is a tender steak?” You see, there is every chance that had Mike given the steak to a different person to eat, they would have come to a different conclusion. He could have had 5 people try his marinated steaks and non-marinated steaks and gotten results all over the place. And what if, as he added more people to the experiment the harder it became to pin any definite results to the testing? This is assuming he had more than one participant, which is doubtful.
What is tenderness? This is the operating question.
This is the first thing the Mythbusters had to do. They had to figure out just what tenderness is and how to measure it. According to Adam Savage, in fact, they spent a whole day testing and ended up scrapping it. These tests, which were never shown on the finished episode, ended up being a wash because the Mythbusters found they were using the wrong parameters. If the Mythbusters had a hard time with the tests, what are we to assume about Mike at home in his kitchen, laptop at hand?
It turned out that the USDA (United States Department of Agriculture) already had a way of testing steak tenderness that resulted in accurate and repeatable data: a machine that measures how many pounds of force it takes to punch a hole through the meat. In fact, the Agricultural Marketing Service of the USDA, in cooperation with ASTM International, has developed a USDA Tenderness Program which is meant to help consumers find specific cuts of beef which meat standards allowing the claims USDA tender or USDA very tender. To find whether a cut meats minimum thresholds for tender or very tender claims, Warner Brantzler shear force measurements are used. This measures the amount of force needed to shear through a sample of meat. The measurements can then be compared to national standards. The video below shows such a test being performed.
Munching on a steak and writing down your opinion on its state of tenderness does not produce accurate objective data. It produces subjective, and perhaps nonrepeatable, impressions. In fact, you can often surmise when such experiments are not valid. You see, when you perform an experiment, you often already have an idea about what your testing will show. In the case of marinades, our own experience and the experience of many professional cooks should tell us that if marinades do enhance tenderness, the difference is small. If Mike’s results are dramatic and he writes of “the most tender steak he’s ever eaten,” we are right to question his methods, and even his honesty.
Often, dramatic results that go way beyond what most professionals would expect are seen as more valid and as “proving everyone wrong.” Experimental results are rarely so dramatic, however. It is usually difficult to tease out the small differences between a test and a control. Tons of positive results, in fact, often leads to revelations of data manipulation, salami-slicing, and other malfeasance on the part of scientists. Research the recent fall from favor of famous food researcher Brian Wansink, for example.
Tenderness is something that can be accurately measured. Many other aspects of food can be accurately measured as well. Yet, most informal food experiments don’t even bother to define what they are actually measuring. You can cook a steak in a skillet, on the grill, or in the oven. You can use a combination of these. But, an experiment designed to test “which steak is best” will never yield dependable results. Why? Because the results rely on the subjective opinions of tasters and those results are not repeatable.
The Cola Wars
Do you like Coca-Cola or Pepsi? Pepsi used to run television commercials showing people engaging in blind taste-tests. They claimed more people chose Pepsi than Coke. This was never true, but even if it were, would these experiments prove that “Pepsi is better than Coke?” Of course not. It is a matter of preference. If you like Coca-Cola better than Pepsi then it is better to you, and that is all there is to it. If you think steak is any different, you’ve been watching too many cooking shows on television. The steak which is better is the steak you like best. Even if it is not considered “correct” by certain standards. As well, it is possible that you might enjoy a Coke sometimes but at other times want a Pepsi. I’m one of those odd ducks, in fact. The same could be true of steak, which may be even harder to pin down than cola preferences.
What Defines a Good Steak?
Yes, we can assume that certain standards define a good steak. And we can assert that more people will enjoy a steak that meets these standards. But does this prove it’s better? There is, after all, no right or wrong answer to the question which steak is better. So, all tasting can do is indicate preferences, and only then if the same tasting experiments are performed with a sufficiently large number of participants. The results from one or two tasters is no result at all.
Sensory Evaluation and Objective Evaluation
Food scientists grapple with this question. Both sensory evaluation and objective evaluation on food quality are important to food science. Sensory evaluation is not objective. Therefore, it requires many, many participants. Having people sit down to compare two different steaks and rate their tenderness would fall under sensory evaluation. The number of people required to produce meaningful results makes such experiments time consuming and very expensive. And even then, the results would simply show that one or the other would tend to be more pleasing to consumers or diners.
And then, if you performed the same testing again, lo and behold, you might end up with different results. This is because sensory evaluation, relying on inaccurate and changeable human perceptions, is highly variable and thus difficult to repeat.
The machine that the Mythbusters built to emulate the USDA machine for testing tenderness, however, would fall under the objective evaluation umbrella. They spent around $50 to make their machine and could have tested many many samples in one day with dependable and repeatable results.
However, here we encounter a problem. You may have been asking yourself one very pertinent question: Just because a machine says a steak is more tender, does this mean it will be more enjoyable?
The answer, is, of course, no. Objective evaluation of food only measures ONE thing at a time But, if you were a company trying to enhance the tenderness of your packaged meat product, you would need to use objective evaluations to accurately narrow down your choices and then rely on human participants to tell you which methods produced the more acceptable product.
So, in reality, whether a steak is objectively tender is not the same question as to whether it will be more acceptable to a majority of people who eat it. And, since it is difficult to even ensure uniform cooking techniques and small variables can produce recognizable changes in the finished product, it would be difficult for a home cook to approach the kind of scientific rigor of a food company employing trained food scientists.
The Mythbusters did indeed “confirm” that explosives could be used to tenderize a steak. They also found that steak could be tenderized in a clothes dryer.
And this, I hope, answers the question for those readers who doubt I have considered all the sources when I “bust a myth” or attempt to evaluate it. I also hope it makes you a bit more skeptical of those who claim to bust cooking myths by testing them. You’ll notice that I do not, and I think you can see why now.
Outfits like America’s Test Kitchen and some other oft-mentioned sources can be entertaining and very informative, but they are not what I consider valid scientific sources. They often do not have any means of objective evaluation, and have far too few participants for any reliable standard of sensory evaluation. They also, as well, fail to operationalize. Many questioned my advice that it is not really practical to dry age your own steaks at home. They told me they had done it themselves, with great success. Those steaks were good! Well, what is good? How do you operationalize the concept of a good steak?
In the end, what we are left with is the opinion of one or a few cooks who tried two or three different ways of cooking something. If you try the experiment yourself, your results may vary! While food bloggers are concerned with busting cooking myths, food scientists are concerned with improving food quality, shelf-life, and consumer satisfaction, among other things. Often, however, truly rigorous experimentation has been performed and a dedicated search can often turn up published results from scientific journals.
Many other food and cooking myths are not put to rest based on cooking experiments but based on explanations of the underlying science. If a certain cooking fact or ingredient reaction seems to violate known properties, we can often, through relying on scientific explanations that themselves rely on other sources of evidence, bust a myth without having to pick up a knife or turn on a stove. We can never claim to have proven it, however, until a proper experiment is done. When the underlying natural laws are fairly clear, as they often are, it may not be worth getting into a tizzy over, unless you simply enjoy experimenting with food. Just realize that your results will probably be subjective.
This article contains one or more Amazon affiliate links. See full disclosure.