Living at the Extremes in the Information Age

As a change of pace, this issue's column isn't devoted to a cutting edge technology or process. Instead, we will explore a recent book, The Black Swan, The Impact of the Highly Improbable, by Nassim Nicholas Taleb (Random House, New York, 2007).

Paul Bleicher MD, PhD

Taleb's terminology

Most people lead their lives assuming most observations, events or measures are distributed in a bell-shaped normal distribution. Of course, they probably don't even realize their assumption or even explicitly understand the normal distribution. Yet most of us expect and act as if events will unfold for us (and the world) roughly within the bounds of our limited experience and information gathering. These assumptions bring organization and comfort to our lives, but they may prevent us from avoiding catastrophic events as well as benefiting from spectacular opportunities.

Consider your daily commute to work. You have likely developed a comfortable sense of the pattern of traffic and drive "defensively," prepared to act on the wide range of bad driver behavior you might expect. In the eyes of Taleb, you are living in "Mediocristan," where observations fall within a normal distribution of your experiences, and no individual event has a meaningful impact on your life.

Suddenly, one morning you are faced with a very fast moving and erratic car hurtling directly toward you. A wrong lane driver is a completely unexpected and unpredictable event, so far off the normal curve that we might never have considered it possible nor worth thinking about. In fact, we had always been living (and driving) in "Extremistan," where highly improbable and infrequent events dominate in their impact on our lives. Most of us simply don't realize that we are.

Our wrong lane, out of control driver is an example of a "black swan," as described by Taleb: an unexpected and unpredictable event that is so far from our aggregate experience that it has an effect far out of proportion to others.

According to Taleb, most of us live our lives assuming that we are in Mediocristan when, in fact, we are always in Extremistan, where that unpredictable black swan can happen at any time.

Taleb is the latest in a series of commentators who are thinking about the impact of nonGaussian distributions, particularly the power distribution, which model the way that infrequent events can have disproportionate impacts. He isn't just talking about number crunching or investments (where he has used these principles professionally), but in almost every aspect of our lives. It is worth a very general understanding of the power function before delving into its impact.

Power laws and populations

The 80/20 rule (which is actually quite arbitrary and could be the 75/25 rule or other skewed fraction) actually fits a simple, mathematically defined law, known as the power probability distribution. If the normal curve resembles a bell, the power distribution resembles a hockey stick—the Y axis starting at its highest point at some X and rapidly descending to a long tail.

Vilfredo Pareto (1848–1923) used the power distribution when studying the distribution of wealth in a society (still one of the most easily understandable examples, especially in contrast to a normally distributed measure). Many attributes of the human population are normally distributed, such as weight, height, and intelligence. The heaviest, tallest or most intelligent person you could think of, if lumped in with 100 randomly selected people, would barely budge the median value. However, the personal wealth of these same people is quite another thing.

If we take the richest person we can think of (for example, that entrepreneur/philanthropist from Seattle with a net worth estimated at $56 billion dollars), his overall effect on the median would be staggering. In fact, it would shift the median up by $560 million dollars! Obviously, in sampling a population, medians of personal wealth have little meaning. Of course, you could argue that it would be highly unlikely for a random sample to include Bill Gates, but that is beside the point. Bill Gates is a highly improbable black swan when it comes to wealth.

The power distribution is, in fact, a model of many different things. An often sighted example includes the frequency of words in books (a small number of words account for most text, and then many, many more are used occasionally).

Thinking in normal curves

There are many reasons why most of us live in Mediocristan and are accustomed to thinking in Gaussian terms. To begin with, life has become exceedingly more complex than it was in the past, when humans developed their patterns of thinking. We tend to extrapolate from small numbers of observations, rather than the large number necessary to sample a complex environment. We are comforted by a consistent pattern of observations, confident that we can depend on them to continue ad infinitum.

We are also taught from a young age to average observations without considering the distribution of what we are measuring. We consciously and subconsciously discard outliers as mistakes or incorrect observations and consistently make too few observations to sufficiently model outcomes. We fall prey to what Taleb calls the Ludic Fallacy: that we can use an observation-based model of reality (or "game") to make predictions (and choose actions) in our lives based typically on an expectation of a normal distribution of possible outcomes.

Our struggle to make sense of the enormous and growing volume of data and information leads to two important adaptations that also keep us from seeing the extremes: confirmation bias and storytelling.

Confirmation bias is simply the picking and choosing of information that confirms a pre-existing or developing bias. In a sea of information, our minds tend to seek out and emphasize that information which confirms our world views. In addition, our intense desires to understand and explain the world leads us to draw correlations where none may exist. Since positive correlations (for example, in clinical research) are notable, publishable, and fundable—and a lack of correlation is not—we work our way toward a world view that fails to represent its true complexity.

Another important adaptation is the universal tendency to weave stories that explain current reality as a sum of past events. For example, historians explaining how we created computers or entered World War II weave a story that describes the events and milestones that form a meaningful progression toward these outcomes without including the thousands of false steps and (seemingly) meaningless diversions. These stories typically leave out the wholly unexpected, serendipitous events that were responsible for most of the progress. Outcomes are generally determined by black swans.

Admitting we don't know

The future will be shaped by black swans, and by their very nature they are unpredictable. Even if predictions may be seemingly correct in the short or medium term, a single black swan can make them worthless, and so they are generally worthless from the start (unless they are correct by shear accident, like a broken clock that tells the correct time twice a day).

If black swans are impossible to predict, and predictions are generally worthless, how can we plan to maximize our advantage and minimize our downside exposure to these events? First, we need to be willing to say that we simply don't know when we don't—which is most of the time when predicting future events. And beware of those who do make predictions.

Observe, question, form intuitions, and observe some more. Take actions based on observations, not on theory. In other words, it is ok to assume that the sun will rise tomorrow morning, but don't put all your eggs in the basket of, say, globalization of outsourcing just because somebody smart predicted that it would be "the future."

Taleb argues that we should choose courses in our careers or lives that expose us to positive black swans and not negative ones. Working on projects and areas that have little upside (he cites catastrophe insurance and homeland security as two examples) exposes you mostly to the downside of black swans. However, investing time, money, and resources in areas where a black swan could bring windfall results is definitely worthwhile.

For example, spend 85% of your time on projects or investments that are rock solid, stable, and have guaranteed returns, albeit small. Then, take the remaining 15% for highly speculative bets, projects or investments with very large payoffs. This creates a floor to the downside, but exposes you to the potential for the large gains achieved by those rare positive black swans that come from the 15% of speculation.

Finally, positive black swans don't come along every day. When an opportunity—or something that looks like an opportunity—arises, seize it. We shouldn't wait for these opportunities to drop into our laps either, but we should actively go out and seek them.

Paul Bleicher MD, PhD, is the founder and chairman of Phase Forward, 880 Winter Street, Waltham, MA 02451, (888) 703-1122, paul.bleicher@phaseforward.comwww.phaseforward.com. He is a member of the Applied Clinical Trials Editorial Advisory Board.