One man randomized his life. He teaches us why we need much more trial-and-error experimentation—in our lives and in public policy. Here's how it might work.
Brian, another brilliant discussion! Your article begs the question of why people do not explore, experiment, or take to different experiences? Sure, “knowingness” is a real issue, but I hypothesize it goes beyond that. I think the answer lies in the underlying fear of change and differences. But when does fear override the necessity for exploration and change? I think Tom Nichols at The Atlantic has some good thoughts. It is being too comfortable and having no need to push to explore and open up the world. It is living in a bubble faced with things one “knows” leading to “knowingness” and not realizing how much one does not truly know or understand. It is an inherent lack of curiosity. It is a need to feel in control at all times, even when the reality is control is an illusion.
Evolution will eventually weed out these traits, but how long will it take? Can we as a society afford to wait for that? After all, we are seeing the trend of making education “comfortable” and sanitized so as not to traumatize those who may be shocked by the ugliness and reality of the past, who fear science because they do not understand it, and are beating the experimental and exploratory instincts out of our children.
If we start with anything in public policy, we start by encouraging exploration and experimentation in our educational experience rather than rote memorization requires to take tests as is the case in the US today. Require exploration through narrative and writing, not multiple choice tests.
Thoughtful comments, Paul! And don’t think that just because I write about it that I’m some arch-experimenter constantly. I am guilty of these traits too, but I’ve realized in the last several years that it’s a lot more rewarding to branch out.
I agree with you about education, though amazingly, I think the US is comparatively good at that internationally. The education system in China, for example, certainly does not foster quite so much creativity and questioning of received wisdom, which is often the intellectual basis of exploration.
3. "It's not because things are difficult that we dare not venture. It's because we dare not venture that they are difficult." -- Seneca. Thank you Brian for this!
Here's a different kind of incuriosity: Often we naturally or semi-formally set things up as if they were experiments and we have access to the results, but we don't use the info because we forget that we had been curious in the past and that we can change our ways in the future. I'm remembering a job I had years ago. We built a particular kind of website for businesses to sign up their new customers, and we could customize it a little for each business. At the outset, we didn't know whether Design A or Design B would push more customers to choose a certain option, so we let each business implement whichever design they felt was more aesthetically pleasing. Turned out, Design A was hugely more effective, leading to something like 10x more selections. We had this firm data. We should have just stopped offering Design B at all and assured businesses that the only design we continued to offer had been "tested and proven." Yet we continued to offer both options and to allow businesses to pick the useless one, as if we didn't have a decade of data telling us which was better. I think it was because the experiment was never formal. There was never an end date by which anyone was responsible for running a report and drawing a conclusion. Our marketers became accustomed to pleasantly offering businesses "a choice," and by the time we'd collected years of data, no one remembered that we had the means to determine objectively which choice was better. I would sometimes raise this topic, and people treated my suggestion as though I were proposing a new experiment or some kind of extra work, although I was just pointing out that historically there was 10x more money through Door A than Door B.
Great story, Tucker - and yes, I think this is definitely true. A ton of people create quasi-experimental data but never share it, and there are a lot of people who try things out in similar ways but never communicate.
I think that experimental policy would still suffer from some of these problems; Republicans wouldn’t believe data from an experiment led by a Democratic trial, and so on, but it’s best to try things out and offer the info. Perhaps the analogy in your story would be to offer both designs but give the businesses the clear data that shows one design outperforms the other tenfold.
I have a smart friend who says that all government policies and regulations should have funding built in to analyze whether they're actually achieving their stated goals. Which implies that all of them should have stated goals that are measurable.
While we root for more experimentation in the public policy realm (and more generally, life), it's important to recognize that experimenting needs autonomy to be creative and the acceptance of 'failure' in the form of failed experiments.
Measuring experimental results - a step largely forgone in policy making in a most of the world is a necessary precursor for experimenting. We need to also be able to measure success and failure of policy makers differently and not tie them to success or failure of the experiment. When failed experiments inevitably occur (by design), they need to be seen in the context of the larger problem solving effort. Also taking a further cue from evolution, weeding out bad experiments - I.e stopping starting and course changing - should be an acceptable thing to do, provided the reasons are the right ones.
Indeed, we need to choose policymakers differently - selecting for ability to experiment well and make process modifications rather than for ability come up with the right policy and make outcome guarantees.This is a more fundamental shift than what the article gives it credit for. Even small companies struggle violently with it.
I agree with the need for more experimentation though.
I'm a little stuck on the premise that "most laws are based on hunches". Don't we send out surveys, solicit comment, debate the options, etc.? At some point a decision has to be made and while that decision may rely on incomplete data and be based in compromise, it seems to me that most being based on a hunch would be hard to support.
Most laws are proposed based on someone’s idea of what will work, often without any form of testing or verification, and because they take so long to pass and are often in a vortex of partisan polarization, the feedback loops that lead to self correction often don’t work. For example, the UK only has three income tax brackets. Why? Why not 12? Why not 7? It’s arbitrary and alternatives aren’t tested. With path dependency and lock-in, a policy idea can last for decades, even if it’s not optimal and may actually be harmful. So it’s not that it’s not debated, but rather that there’s not rigorous testing in virtually any policy making - and very limited checks to decide what, if anything, constitutes the “best” option vis a vis any objective evidence.
Since it would seem that an awful lot of proposed legislation comes from special interest origins, hunch still doesn’t seem to me to be the best descriptive. Maybe if the “hunch” includes what the special interest determines will work best for them. Regardless, since there is no shortage of competing special interests, one would think that critical examination of the data and testing each used to support their proposed legislation, rigorous or not, could at the very least show the paths that the data does or doesn’t support.
I should point out that my musing comes not from an expert but from simply a curious observer. My observation though is that it seems that the driver of disruption for the path dependency and lock-in you mention is going to be psychological. There seems to be no shortage of great concepts that have fallen victim to the ego. Elon Musk didn’t build Tesla on peoples loyalty to the commons. He built it on satisfying peoples drive for status. If we are to preempt the Darwin Award that closes out the Anthropocene my view is that it is going to have to be in solutions that feel empowering.
I think the problem is that special interests are often not trying to solve a social problem so much as protect a core interest of a smaller group. So, instead of asking a question like “which policy reduces gun deaths the most?” You instead have a special interest that is asking a different question: “which policy ensures that gun owners are happy?” Or “which policy allows gun manufacturers to make the most money?” Now, those are questions that are easier to answer without experiments, whereas the first question is based a bit on what I call hunches, because it’s difficult to know for sure without testing. The problem, though, is that some laws only work once they apply to everyone (the gun example shows this) as gun laws in a village next to a city will have little to no effect if people can just buy bazookas in the city.
So yes, there are wrinkles, but such is the nature of essays on here - there’s a limited word count!
I suppose “special interests” is more a misnomer and “entrenched interests” may be more accurate. My concept of them actually competing on the basis of research is likely wishful (make that fantastical) thinking. Much appreciate your work and response.
Yes, there is a wide gulf between groups that employ lobbyists and those who study policy effectiveness. I also speak from personal experience as the former Policy Director for a successful campaign for Governor of Minnesota (we won). My hunches about what would work best were part of what made policy real. I proposed an idea, we discussed it, it became part of the platform, some of them became laws.
Again a timeless and timely elegant essay. Thank you, Brian.
Brian, another brilliant discussion! Your article begs the question of why people do not explore, experiment, or take to different experiences? Sure, “knowingness” is a real issue, but I hypothesize it goes beyond that. I think the answer lies in the underlying fear of change and differences. But when does fear override the necessity for exploration and change? I think Tom Nichols at The Atlantic has some good thoughts. It is being too comfortable and having no need to push to explore and open up the world. It is living in a bubble faced with things one “knows” leading to “knowingness” and not realizing how much one does not truly know or understand. It is an inherent lack of curiosity. It is a need to feel in control at all times, even when the reality is control is an illusion.
Evolution will eventually weed out these traits, but how long will it take? Can we as a society afford to wait for that? After all, we are seeing the trend of making education “comfortable” and sanitized so as not to traumatize those who may be shocked by the ugliness and reality of the past, who fear science because they do not understand it, and are beating the experimental and exploratory instincts out of our children.
If we start with anything in public policy, we start by encouraging exploration and experimentation in our educational experience rather than rote memorization requires to take tests as is the case in the US today. Require exploration through narrative and writing, not multiple choice tests.
Thoughtful comments, Paul! And don’t think that just because I write about it that I’m some arch-experimenter constantly. I am guilty of these traits too, but I’ve realized in the last several years that it’s a lot more rewarding to branch out.
I agree with you about education, though amazingly, I think the US is comparatively good at that internationally. The education system in China, for example, certainly does not foster quite so much creativity and questioning of received wisdom, which is often the intellectual basis of exploration.
Great one! Was Hawkins inspired by the 1971 novel Dice Man (https://en.wikipedia.org/wiki/The_Dice_Man)?
I came across that but haven’t read it!
3. "It's not because things are difficult that we dare not venture. It's because we dare not venture that they are difficult." -- Seneca. Thank you Brian for this!
Here's a different kind of incuriosity: Often we naturally or semi-formally set things up as if they were experiments and we have access to the results, but we don't use the info because we forget that we had been curious in the past and that we can change our ways in the future. I'm remembering a job I had years ago. We built a particular kind of website for businesses to sign up their new customers, and we could customize it a little for each business. At the outset, we didn't know whether Design A or Design B would push more customers to choose a certain option, so we let each business implement whichever design they felt was more aesthetically pleasing. Turned out, Design A was hugely more effective, leading to something like 10x more selections. We had this firm data. We should have just stopped offering Design B at all and assured businesses that the only design we continued to offer had been "tested and proven." Yet we continued to offer both options and to allow businesses to pick the useless one, as if we didn't have a decade of data telling us which was better. I think it was because the experiment was never formal. There was never an end date by which anyone was responsible for running a report and drawing a conclusion. Our marketers became accustomed to pleasantly offering businesses "a choice," and by the time we'd collected years of data, no one remembered that we had the means to determine objectively which choice was better. I would sometimes raise this topic, and people treated my suggestion as though I were proposing a new experiment or some kind of extra work, although I was just pointing out that historically there was 10x more money through Door A than Door B.
Great story, Tucker - and yes, I think this is definitely true. A ton of people create quasi-experimental data but never share it, and there are a lot of people who try things out in similar ways but never communicate.
I think that experimental policy would still suffer from some of these problems; Republicans wouldn’t believe data from an experiment led by a Democratic trial, and so on, but it’s best to try things out and offer the info. Perhaps the analogy in your story would be to offer both designs but give the businesses the clear data that shows one design outperforms the other tenfold.
I have a smart friend who says that all government policies and regulations should have funding built in to analyze whether they're actually achieving their stated goals. Which implies that all of them should have stated goals that are measurable.
While we root for more experimentation in the public policy realm (and more generally, life), it's important to recognize that experimenting needs autonomy to be creative and the acceptance of 'failure' in the form of failed experiments.
Measuring experimental results - a step largely forgone in policy making in a most of the world is a necessary precursor for experimenting. We need to also be able to measure success and failure of policy makers differently and not tie them to success or failure of the experiment. When failed experiments inevitably occur (by design), they need to be seen in the context of the larger problem solving effort. Also taking a further cue from evolution, weeding out bad experiments - I.e stopping starting and course changing - should be an acceptable thing to do, provided the reasons are the right ones.
Indeed, we need to choose policymakers differently - selecting for ability to experiment well and make process modifications rather than for ability come up with the right policy and make outcome guarantees.This is a more fundamental shift than what the article gives it credit for. Even small companies struggle violently with it.
I agree with the need for more experimentation though.
I'm a little stuck on the premise that "most laws are based on hunches". Don't we send out surveys, solicit comment, debate the options, etc.? At some point a decision has to be made and while that decision may rely on incomplete data and be based in compromise, it seems to me that most being based on a hunch would be hard to support.
Most laws are proposed based on someone’s idea of what will work, often without any form of testing or verification, and because they take so long to pass and are often in a vortex of partisan polarization, the feedback loops that lead to self correction often don’t work. For example, the UK only has three income tax brackets. Why? Why not 12? Why not 7? It’s arbitrary and alternatives aren’t tested. With path dependency and lock-in, a policy idea can last for decades, even if it’s not optimal and may actually be harmful. So it’s not that it’s not debated, but rather that there’s not rigorous testing in virtually any policy making - and very limited checks to decide what, if anything, constitutes the “best” option vis a vis any objective evidence.
Since it would seem that an awful lot of proposed legislation comes from special interest origins, hunch still doesn’t seem to me to be the best descriptive. Maybe if the “hunch” includes what the special interest determines will work best for them. Regardless, since there is no shortage of competing special interests, one would think that critical examination of the data and testing each used to support their proposed legislation, rigorous or not, could at the very least show the paths that the data does or doesn’t support.
I should point out that my musing comes not from an expert but from simply a curious observer. My observation though is that it seems that the driver of disruption for the path dependency and lock-in you mention is going to be psychological. There seems to be no shortage of great concepts that have fallen victim to the ego. Elon Musk didn’t build Tesla on peoples loyalty to the commons. He built it on satisfying peoples drive for status. If we are to preempt the Darwin Award that closes out the Anthropocene my view is that it is going to have to be in solutions that feel empowering.
I think the problem is that special interests are often not trying to solve a social problem so much as protect a core interest of a smaller group. So, instead of asking a question like “which policy reduces gun deaths the most?” You instead have a special interest that is asking a different question: “which policy ensures that gun owners are happy?” Or “which policy allows gun manufacturers to make the most money?” Now, those are questions that are easier to answer without experiments, whereas the first question is based a bit on what I call hunches, because it’s difficult to know for sure without testing. The problem, though, is that some laws only work once they apply to everyone (the gun example shows this) as gun laws in a village next to a city will have little to no effect if people can just buy bazookas in the city.
So yes, there are wrinkles, but such is the nature of essays on here - there’s a limited word count!
And good point about Tesla!
I suppose “special interests” is more a misnomer and “entrenched interests” may be more accurate. My concept of them actually competing on the basis of research is likely wishful (make that fantastical) thinking. Much appreciate your work and response.
Yes, there is a wide gulf between groups that employ lobbyists and those who study policy effectiveness. I also speak from personal experience as the former Policy Director for a successful campaign for Governor of Minnesota (we won). My hunches about what would work best were part of what made policy real. I proposed an idea, we discussed it, it became part of the platform, some of them became laws.
It's funny but not great.