Le vrai est trop simple, il faut y arriver toujours par le compliqué.
The truth is too simple: one must always get there by a complicated route.
Letter from George Sand to Armand Barbés (1867)
We kicked off this series with a bold objective: learning how to live full of both scientific skepticism and the wonder of discovery. With clear eyes and full hearts. We will do this best, I wrote, by identifying and rooting out sources of bias and systematic error wherever we find them in our thinking and research.
This installment was going to be about how we permit the intrusion of bias into the very questions we ask. I was halfway through writing it when I realized that there was still more we needed to talk about first. Because whether our answers become biased in our writing down of questions, in our thinking very hard about the answer, or in our writing down that answer, the sources of the systematic errors which cripple our thinking are themselves often predictable and consistent.
We’re on a journey of discovery about the nature of discovery, you and I, and we need to make a detour.
Careful, though, or you’ll miss the turn.
A little more than an hour after you leave Austin, your GPS will tell you to turn right. A split-second later, your brain will retort, ‘There is no way that rough, barely two lane, curbless road with a double-wide on the corner is ‘Main Street.’ Sorry, brain. It is…or, was. It’s a small town. It’s also Saturday, which means that the cattle auction is taking place at the livestock commission. If you hear mooing, you will know you went too far. The beef you’re looking for is of a different sort, and once you have righted yourself on Main Street, you aren’t likely to miss it. Even at 7:45 AM – yes, sorry, did I mention that it’s basically open for four hours on Saturday morning? – the double-parked cars and the line of weary travelers bearing Buc-ee’s growlers full of coffee shall be a sign unto you.
We have long since come to expect that our best regional cuisine will often come from humble, out-of-the-way places. There’s a reason the Michael Scott bit about Sbarro’s in The Office goes over so well. The hipster meme – “Oh, it’s a weird, out of the way little place – you probably haven’t heard of it” – is already five years stale at this point. We’re all in on the joke now. So if I told you that the best cut of smoked meat in the world is beef brisket, and that the best beef brisket in the world comes from a little place in Lexington, Texas that’s only open on Saturday mornings, you probably wouldn’t bat an eye. Especially since it has now been at the top of the Texas Monthly list for more than a decade.
But I’m telling you, everything about Snow’s BBQ is wrong.
For starters, it really is just a one-day-a-week operation. It is extraordinary enough (and popular enough) that it could do a bustling business on most days like the joints in Austin or Lockhart. But despite the fixed cost-related challenges of a Saturday-only approach, they haven’t made the switch. The owner of the place is a former prison guard and rodeo clown whose day job for most of the time he has owned Snow’s was at a coal mine. The pitmaster is an 83-year old former butcher who works most of the week in maintenance at the high school down in Giddings.
All that makes for a good story. But that’s not what’s wrong. It’s the way they BBQ here. Correct BBQ is about indirect heat. Tootsie Tomanetz cooks almost everything over direct heat, and I think if she had her way, would still be doing it on briskets, too. Correct BBQ is about low and slow. Tootsie Tomanetz cooks several cuts – including a really excellent sausage – at much higher temperatures than the typical joint. Correct BBQ is about sourcing bespoke prime-grade or American wagyu beef from idyllic ranches in Montana. Tootsie Tomanetz buys her beef from a butcher in Taylor called O’Brien Meats that doesn’t even have a website. Correct BBQ is about washing the meat in a constant billow of smoke. Tootsie Tomanetz’s fires are heavy on hot coals and light on fresh logs – a much less smoky fire. Correct BBQ doesn’t rely on shortcuts like the Texas Crutch. Tootsie Tomanetz has been wrapping her briskets in foil for years. Oh, and by the way – Correct BBQ is a guy thing. I guess no one told Tootsie about that one either.
If it sounds like Correct BBQ is a religion, that’s because it is.
One Thin Red Line
If you want to understand the religion of Correct BBQ, there is no greater symbol of it than that the pale red strip at the bottom of this slice of brisket.
That little strip of color is called a smoke ring, and it is a fundamental part of the lore of Correct BBQ. Restaurants around the US frequently tout it as an indication of properly smoked meats. The largest national body governing competition BBQ standards, the Kansas City Barbeque Society, included it for many years among its formal judging criteria. Despite its removal some years ago, many judges still swear by it, or at a minimum acknowledge the subconscious effect it has. Like this one. And this one. Even though most competitions today don’t formally recognize it as part of the judging standard, it remains an obsession of most aspiring and backyard cooks.
The most common reason given for celebration of the smoke ring is a tautological one: It is the hallmark of Correct BBQ. Perhaps one layer below a pure tautology, the smoke ring ‘is believed to show that you have done a good job and properly low and slow smoked the meat in question.’ In other words, the smoke ring is accepted by many as post hoc evidence of proper technique, and two aspects of the technique in particular: cooking meat slowly over low temperatures, and cooking it over a smoky wood fire. It is a beautiful bit of lore that adds romance and an air of artistry to an otherwise (literally) visceral activity. This pink flesh is the result of a lazy fire tended dutifully, with billowing smoke slowly washing over a well-seasoned cut of meat over a period of hours. As such romantic lore tends to be, the smoke ring was for many years ingrained as common knowledge among aficionados – a thing that everybody knew that everybody knew. It was the answer from the gods to him who performed the ritual properly and with a pure heart.
If only it were true.
The smoke ring is not ‘smoke penetrating the meat.’ It is not even evidence of a significant quantity of smoke. It is the result of a chemical reaction between nitric oxide and myoglobin, the main non-water substance inside the ‘juices’ in a piece of meat. The size of a smoke ring in a piece of meat is determined entirely by the quantity of these gases that come in contact with that myoglobin before it hits about 170 degrees. The presence of those gases has only a limited relationship with the quantity of ‘smoke’ produced by the cooking fire. You can produce comparable quantities of those gases with plain old charcoal briquets. If you’re pressed for time, sprinkle that brisket with curing salts containing sodium nitrite and throw it in the microwave. You’ll be the lucky owner of a disgusting hunk of gross with an exquisitely deep salmon smoke ring.
Even the preference for low-and-slow cooking is a methodological abstraction of the scientific process of collagen denaturation and breakdown. It works, but not because of some direct relationship between flavor and the speed of cooking, but because the technique strikes a balance between maintaining high enough internal temperatures for long enough to effectively facilitate beneficial chemical processes in intramuscular collagen on the one hand, and minimizing excessive drying and evaporation on the meat’s exterior on the other. For many cuts, each of these processes can be achieved through a higher temperature cook and a longer period of insulated resting of the prepared meat.
Likewise, the disdain many had for techniques like the Texas Crutch – wrapping a brisket during part of its cooking process – was based on a belief that it replaced smoking with steaming (which is not entirely incorrect), and that it sped up the cooking in a non-traditional way that would harm the product (which is nonsense). We now know, of course, that wrapping a piece of meat reduces the process of evaporative cooling and results in moister, more flavorful BBQ.
The evaluation of food is a subjective, human thing. But that is the point. In any field for which the interpretation or objective function – the thing we’ve solving for – isn’t quantifiable or even knowable, any tangible method feels like a godsend. Won’t someone just tell me what to do? But that method will always be an abstraction from the thing we seek. When these abstractions become ritual, the risk is that our process of discovering facts about that topic will be guided by its relationship to the abstraction, to the religious myths that we memorize and pass along to others.
This is all made more difficult by the fact that there may be good reasons for parts of the ritual. I still personally have much better luck, for example, cooking most things at a very low temperature over a very long period. The point is that the ritual of Correct BBQ stifled the exploration of newer, better ways to prepare it. People made BBQ to most closely resemble what they expected from the ritual. In algorithmic terms, we were stuck in a local optimum and needed enough crazy-ass ideas to succeed to have any hope of achieving movement in our literal and figurative posteriors. Sure, there were always exceptions and independent thinkers. But it has really only been in the last two or three decades that people like Tootsie Tomanetz who didn’t give a damn what anyone else thought have come into the mainstream. It isn’t that Tootsie, or Aaron Franklin or any of the other demigods of Texas BBQ aren’t respecters of tradition. These are post oak-only, salt-and-pepper purists, after all. It’s that their experiments weren’t guided by hewing to the rituals of Correct BBQ for the sake of those rituals.
Snow’s BBQ IS Tannu Tuva, y’all. Not literally, I mean, although it is a pain in the ass to get there. I mean that it is proof that some of the world’s greatest joys come from unearthing beauty that remains beautiful even when we discover what it really is. Tannu Tuva wasn’t something that Richard Feynman feared would become less beautiful or magical through its discovery any more than beautiful food would cease to be art because we understand the science behind its flavors, textures and aromas. Knowing only adds.
I have a friend who’s an artist, and he sometimes takes a view which I don’t agree with. He’ll hold up a flower and say, “Look how beautiful it is,” and I’ll agree. But then he’ll say, “I, as an artist, can see how beautiful a flower is. But you, as a scientist, take it all apart and it becomes dull.” I think he’s kind of nutty. … There are all kinds of interesting questions that come from a knowledge of science, which only adds to the excitement and mystery and awe of a flower. It only adds. I don’t understand how it subtracts.What Do You Care What Other People Think, by Richard Feynman (1988)
Knowing only adds, that is, unless what we most desire is the sanctity of the ritual. But ritual isn’t the only potential enemy of a worthy process of discovery. We must also grapple with the way in which systematic errors and bias creep into our analysis when relevant facts ARE knowable, when some of them ARE measurable…and when they appear to clearly support our theories and priors. As it happens, the field that made Feynman famous not only gives us perhaps the greatest simultaneous source of skepticism and wonder in all of physics, it also deals with exactly this problem. And it just so happens to be about a different kind of thin red line.
Another Thin Red Line (or Two)
What you see below is a stylized illustration of the visible light portion of hydrogen’s emission spectrum.
OK, I admit that I escalated quickly from a discussion of coagulated meat juices, so let’s keep it simple. These are the colors of light emitted when a hydrogen atom moves from a high energy state to a lower one. This was a big deal in the late 19th and early 20th centuries, not just because we were trying to understand electromagnetism, but because observing electromagnetic effects (like, say, light) allowed us to test different theories about sub-atomic particles. It was a beautiful dance between experimental and theoretical physics, between deductive and inductive research methods. In 1916, Niels Bohr built a model that described electrons orbiting the nucleus of an atom at various discrete distances. It wasn’t the first model that gave the atom the solar system treatment, but it was the first that seemed to provide a mechanism explaining the spectrographic image we see above. In other words, Bohr sought a physical description of what rules electrons could be following that could also explain why a change in the energy state of that electron would emit that particular frequency of red light.
He knew at the time that his model wasn’t completely right. While we could observe the effects of what we would later explain using quantum mechanics, we lacked the math and the models to explain those effects. And so the Bohr model relied on quantization heuristics, which is a smarter-sounding way of saying, “Let’s bolt some stuff onto the model we used to use to solve this problem so that it spits out the solution we can observe.” You can think of it like the old up-converters they used to sell for pre-HD cable boxes and DVD players, or the auto-tune on a Selena Gomez record. I’m making it sound more dishonest than it is in service of a joke – a lot of things are figured out by finding out what lies in the gap between our current model and our current observations.
Still, even with Bohr’s quantization heuristic (which defined a discrete list of possible stable states for electrons), the model wasn’t quite right. That’s because while it looks like each of the emitted frequencies is a single line, and while the Bohr model creates a workable physical explanation for that measurement in a hydrogen atom, that isn’t exactly what a spectrograph would measure. If you could look more closely – much more closely – you’d see that there is a fine structure to that red line. In other words, there are two red lines there. The Bohr model didn’t account for this, and not for lack of trying. Enter Arnold Sommerfeld.
Sommerfeld built on Bohr’s model in a few ways. The most obvious change modified Bohr’s framework to one in which the orbits at different energy states were elliptical. Without going into a rabbit hole discussion of angular momentum and phase integrals, the important fact is that Sommerfeld developed a closed form solution that was exactly right in predicting the two red lines for the relativistic hydrogen atom. His was precisely the formula that Paul Dirac would propose for this calculation under full quantum mechanics some twelve years later, and Sommerfeld did so with no understanding of the features of quantum mechanics that were responsible for the fine structure! As L.C. Biedenharn put it in one of many pieces summarizing and exploring the affair, “Sommerfeld’s methods were heuristic (Bohr quantization rules), outdated by two revolutions (Heisenberg-Schroedinger nonrelativistic quantum mechanics and Dirac’s relativistic quantum mechanics) and his methods obviously had no place at all for the electron spin, let alone the four components of the Dirac electron.”
If this wasn’t magical enough, Sommerfeld’s method also left us with the gift of a new dimensionless physical constant for our Standard Model of particle physics, which is a fancy way of saying that we discovered a number that is really important but which isn’t really a measurement or unit of anything. It’s just a number that reflects a fundamental property of the universe. The fine-structure constant, as it is called, can be measured and observed just about everywhere, but cannot be mechanistically explained as a governing rule or force.
It’s 1/137, give or take. Sommerfeld calculated it as the ratio of the velocity of an electron in the first circular orbit of the Bohr model to the speed of light. It’s also the square of the ratio of the elementary charge to the Planck charge. It’s part of the function describing the probability than an electron will emit or absorb a photon. It manifests in the relationship between the energy of a particular photon and the energy level at which two electrons overcome electrostatic repulsion. Or, as Feynman put it:
…[it] has been a mystery ever since it was discovered more than fifty years ago, and all good theoretical physicists put this number up on their wall and worry about it. Immediately you would like to know where this number for a coupling comes from: is it related to p or perhaps to the base of natural logarithms? Nobody knows. It’s one of the greatest damn mysteries of physics: a magic number that comes to us with no understanding by man. You might say the “hand of God” wrote that number, and “we don’t know how He pushed his pencil.” We know what kind of a dance to do experimentally to measure this number very accurately, but we don’t know what kind of dance to do on the computer to make this number come out, without putting it in secretly!QED: The Strange Theory of Light and Matter, by Richard Feynman (1985)
The fine-structure constant is Tannu Tuva, too – a miracle in its discovery, mysterious in its origin, but also measurable. Knowable. Observable. And no less miraculous or mysterious for all that. But it IS the kind of thing you put on your wall and worry about. Not just about the Athena-springing-from-the-head-of-Zeus feeling you get from a number that just happens to be a fundamental identity of the universe. There’s also something unnerving about a property that can be correctly predicted from an abstracted model.
Thankfully, in physics, that unnerving prospect is the exception which proves the rule. After all, it’s not as if physicists stopped trying to better understand quantum mechanics, particle physics and electromagnetism just because Arnold Sommerfeld had figured out what must be happening inside a hydrogen atom. It’s a good thing, too.
In the social sciences, rather less thankfully, the exception IS the rule. Every model we build which seeks to predict some event that is a function of human behavior is nearly always inductively overdetermined and deductively underdetermined. What I mean by that is that investors, journalists, policy wonks and other social scientists can never be as certain as a physicist that our model or analysis reflects a true feature of the world, but we will nearly always have enough observational data to demonstrate to us that it does. We have so many degrees of freedom, so many variables to consider, that with enough data we can usually construct a dozen workable models for why that atom produces those two frequencies of red light. Evidence of this peril is everywhere. It lies in every investment strategy backtest, every interpretation of a politically charged video presented as fact, in every macroeconomic model, and in every perfectly detailed economic report given by a central banker under the aegis of immunizing communications policy.
The credible observer with any measure of experience with the statistical rigor of financial, economic, sociological, psychological and political research will inevitably come to one conclusion: no matter how much we would pretend that it is something else, the vast majority of our research in these fields is heuristic and nothing more. When we treat these data-backed heuristics as part of whatever the equivalent of the Standard Model is in our fields, bias and systematic error will very often be our reward.
The Red Badge of Bias
I believe these two forces – Ritual and Heuristic – are the most constant threats to clear eyes and full hearts throughout our processes of discovery.
Ritual isn’t inherently bad. There are deep parts of us which respond to its pull, and our lives would be emptier if we rejected it fully. We have written about many of the worthy uses of narrative in Holy Theatre, such as the ones brought to bear in the Civil Rights movement or in the Second World War. Yet we must still be mindful of how Ritual steers our questions, our thinking and our answers into right-thinking patterns and convention. It is the source of the monocultural newsroom and the risk-averse investment committee and countless research projects which begin from unproven, unexamined priors. The tyranny of Ritual is its presumption that its priors are self-evident, as morally unworthy of challenge.
Heuristic is also not inherently bad. For example, I believe in functioning markets as spontaneously organized entities that will nearly always defeat a deductive approach to understanding their value. Similarly, there are many non-falsifiable principles whose survival for millennia ought not to be discarded lightly. Where Heuristic imperils our research is in the the post hoc rationalization of our deductive frameworks on a pseudo-empirical basis. Doing so not only directly introduces the risk of systematic error should our inductive process have missed some key latent variable (as it so often does), but it also indirectly shuts off avenues of inquiry and analysis. It is a force of laziness and overconfidence, typified by the belief that once we have ‘proven’ that something is likely under some set of statistical parameters, we are absolved from trying to disprove it further. The tyranny of Heuristic is its presumption that our priors have been proven, and are in no further need of updating.
For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.The Rogers Commission Report (1986)
With apologies to Mr. Feynman, we will need to expand this idea. For a successful news organization, for a successful investor, for a successful technology, reality must not only take precedence over public relations, but over both Ritual and Heuristic. The only way that it is possible for this to happen is through ruthless governance of the priors which influence which topics we take up, which stories we research, and which factors and markets we examine.
And that – following this little detour we’ve taken together – is exactly where we will go in Part 3.