30 September 2018

Einstein’s had a lot of witty and profound sayings, but “The definition of insanity is doing the same thing over and over again and expecting a different result” is not one of them. And yet, these words had often been misattributed to him. It is not exactly clear who came up with the phrase, but it is a fact that a lot of people got convinced that originated behind the famous mustache of Einstein.

He was one of the giants of science in the 20th century, (Einstein, not the ’stache) which means that the phrase has the sound, the ring of something smart. It sounds like something a physics genius might say. The phrase sounds sciencey.

And it had me believing it. Einstein, Meinstein or whoever said it, I subscribed to it. Until my work experience proved it otherwise.

Back when I worked for Twitter, I was running A/B tests. A properly conducted A/B test is one of the closest things you get to science when building a product. It lets you sift the noise and luck out so you can arrive at a piece of knowledge. If I change the product from A to B, I will get X% of my customers better by Y% with statistical significance. That’s the type of sentence you can confidently claim and bet a lot of money on after an A/B test.

And yet, A/B tests lack one crucial property of science. They are not reproducible. If you run an A/B test today, and run the same A/B test six months from now, you might get a different result. Allowing for a different result is emphatically not insanity. I get it, running the same experiment is likely to result in the same effect. But likely ain’t certain.

I saw the same experiment get run twice, the two runs being about a year apart. The results were different. The first time around, the experiment didn’t show any improvement. And yet, half a year later, it did work. The circumstances had changed. What we were trying to test worked better, even though the idea was the same. One year later, it showed positive improvement. It was green.

Same idea. Different result. This is counterintuitive AF. It was surprising after the fact, but even harder to imagine a priori.

Because we tend to overgeneralize from limited experiences and discard ideas that failed once before. Once an idea has been tried unsuccessfully, we tend to give it a black stamp in the passport and ban it from entering our heads again. We want knowledge that we can trust, and it sometimes takes a single foul for us to discredit an idea. One strike. Out.

Unscientific. Because science is all about reproducibility. The definition of science is doing the same thing over and over and knowing that you’ll get the same result. You can prove scientifically that if you drop an apple, in the absence of interference from other forces, it will fall down. It would be incredibly stupid and yes… insane to believe that the apple may not fall down. It will be non-scientific, which means that it is not the way the universe we live in works.

In our universe, gravity force always tries to pull objects down. And science is the way of determining the rules of the universe. But there are limits to science even in a deterministic system.

For example, take magnets. It is reasonable to expect that the North Pole of a compass will always be pointing at the North Pole of the Earth. It has always been pointing the same way. People have used it to navigate around the globe, for centuries, even millennia. And yet, it might flip around. It hasn’t always been that way. The magnetic field of Earth has reversed its direction multiple times, at random intervals


The magnetic reversal of the poles is due to being part of a chaotic dynamic system. Molten metals and whatnot are spinning and twirling and splashing and splashing inside the Earth’s core. If they spin mostly in circles, their electric charge generates a powerful magnetic field. It’s like a cup of coffee spinning after adding milk or sugar, but it is possible to spin it the other way rather quickly by changing the direction in which we stir it with a spoon.

In a chaotic system like that, the further you go out in the future, the lower your ability to predict what would happen. And the catch is that anything could happen. When dealing with chaotic systems, like the weather or politics, “expecting the unexpected” is not insanity. It’s common sense for the long run.

And we are often part of the systems. We adapt, learn new skills, and forget things such as what we had for lunch yesterday. Which means that if we’ve tried to do something before, and it didn’t work, but we’ve changed and we are trying it again… well, we might not be trying exactly the same idea. Not in the scientific sense of it being the same. Of course, if there are a lot of similarities, then we might have a strong prior expectation that the result will be the same. But there always are small differences, and often they matter a lot.

“Try harder” is a phrase many of us have heard, whether from a coach or a boss or a parent. The idea is that the reason for not succeeding before is our level of effort. Some tasks require a lot of strength, concentration and force us to muster our entire self. To “try harder.”

And sometimes the change in the chaotic system is imperceptible to us. We might be in a negotiation and keep on asking our counterpart to agree on something. They might be using our persistence to evaluate how much we care about the topic, and the amount we care might be a factor in them deciding whether to acquiesce to our demands, and by persisting on a topic, we might be more convincing. Such persistence only works sometimes though, as it can be interpreted as badgering, or bullying and could alienate the other side. Even if it gives us a win, it might be a pyrrhic win, dragging us down in the long term.

So… what’s the whole point. Was Meinstein right to point at insanity, where others might see persistence? I think not. I believe unless scientifically proven wrong, we shouldn’t give up easily. To paraphrase another big physicist, David Deutsch “Everything that’s not forbidden by the laws of physics is possible.”

To overgeneralize failure is a logical fallacy. To tell ourselves stories, claiming that things are impossible. These stories are lies.

And we can stop lying to ourselves.

One short post per week, discussing actionable mental models. Join a community of readers, who receive these posts over email.