We are bombarded by messages telling us to worship the gods of efficiency and optimization, life hacking our way to prosperity. It's a trap. Resilience is a smarter, sturdier goal.
An excellent article. I took my elderly, and not very mobile , mother to Kings College Cambridge recently. I was in check list mode- see Kings ( tick); see Trinity (tick); etc. But my mother suggested we just sit down and look at the Kings College court. And observe. And I got a lot more out of the experience.
Great piece! We have experienced this firsthand in medicine….my oncologist husband and I (ob-gyn) as well as children in healthcare live this daily…there is a point of diminishing return in terms of quality of care and ability to be resilient for cases that don’t fit the algorithm. The health system we were part of LIVED this to the extent that an entire hospital was built on their model of efficiency…now dismantled. They had an entire department devoted to their Improvment systems….We wasted weeks in meetings, events, post it notes on the wall with the group think. One delivery model was instituted over 3 years..the staff-physicians, nurses, everyone told them it was a bad idea…but we forged on, now dismantled as well. There was no open door policy to discuss with leadership…they had paid high price consultants to tell them what to do….and so it was….a sad day for the providers and patients. How can you fight this? My husband’s group was independent but sadly market forces and the hospitals desire to capitalize on the chemotherapy profits put them squarely out of business. The “you play in my sandbox or go home” is also part of this. I am certain other industries have experienced this but it is acutely an issue in medicine.
Excellent article. In my career in IT, both programming and as an analyst, we were called on to solve problems. At a large insurance company that I worked for at the time, they were going to measure and improve efficiency on all processes. I argued with my manager that maybe that worked for issuing quotes, but they couldn’t measure my “thought work” and creativity. He said I was wrong. In the end, the company abandoned the measurement initiative, at least in IT. I’ve been through so many versions of this optimization idea, it’s laughable. I learned way too late in life (I’m now semi-retired and work as an aide for my special needs adult child) the lessons from the Reddit thread you posted. But I’m enjoying life so much more now without checklists. Sadly, I believe your message will go unheeded by those in power, at least until it’s too late and the systems break.
I wrote a couple of papers making similar points but never got them published.
More interestingly, and totally by chance (!), I ran across a copy of Fluke at my local library and read it in one sitting. Great stuff! I'll try to write some comments soon
"...despair and the incapacity for leisure are twins..."
"...a certain happiness in leisure...the happiness that comes from the recognition of the mysteriousness of the universe and the recognition of our incapacity to understand it...so that we are content to let things take their course....which Konrad Weiss, the poet, called'confidence in the fragmentariness of life and history'..."
-Josef Pieper, Leisure: The Basis of Culture ( pp. 26-27)
"" Leisure is essential to civilization, and in former times lesiure for the few was only rendered possible by the labours of the many. But theiur labours were valuable, not because work is good, but because leisure is good"
This is excellent. Very inspiring to, so here's a few thoughts/add ons.
1) Notions of resilience make me always think about the Black Death which killed 30, possibly up to 60% (!!!!) of the population of some (most?) parts of mediaeval Europe within a relatively short period. And while it was a cataclysmic event which caused much social upheaval, what I think is remarkable yet not enough remarked on is that *it did not lead to a collapse of the civilisation present in Western Europe at the time*. One in three (or more) people died horrible and scary deaths yet the rest, somehow, went on. And while I know that whole communities descending into "animal barbarity" in times of disaster is to a large extent a myth, the resilience here, not just of individuals or small kin groups but institutions and structures was remarkable.
Obviously some of it was due to a relatively low complexity and centralisation of the "civilization" then. But still.
2) During the few times when I was employed in vastly differing roles, I repeatedly experienced the staffing version of just-in-time "agile" processes: optimising headcounts meant that while everything ticked over fine in normal times, any unpredicted extra work or anyone taking time off for whatever reason meant the organisation or team began struggling. And that often could not be resolved by return to normal, because the normal was optimised for normal. Lack of labour redundancy meant that it was very hard to make up.
3) In addition to sapping joy out of life, it feels like over optimising will reduce creativity and innovation, because one needs the mental spaces of in-between freedom to allow for the hatching and cooking of new ideas.
4) All that said, I feel that the message of "don't live over optimised checklist life" risk something similar to "it's ok to prioritise yourself" message of current pop psychology: the people who will pick it up will be people who really should make up a checklist or two while the ones to whom it's relevant will often add an item of "build in slack into my pomodoro schedule" to their to-do list. I'm saying it as a lazy wanderer who only recently (in my dotage, pretty much) acknowledged that SOME routine and goals are valuable.
"it did not lead to a collapse of the civilisation present in Western Europe at the time" is an interesting point. While obviously there continued to be people and houses and farms and so on, I've seen people suggest that the dramatic increase in power by the surviving labourers indirectly led to economic and political changes that would ultimately give us the end of feudalism and absolute monarchy centuries later. So while society as a whole was resilient, the political system based on serfdom was much more brittle.
Brian, great piece! As somebody who works in power system economics and operation the Estonian case is typical. Power systems are “optimized” subject to constraints and contingencies such as the loss of a large generator or transmission line. Transmission facilities (lines and transformers) are not fully loaded up to account for these contingencies as if we lose a generator or line, the topology has changed and power flows will change and there must be room left on other lines to handle additional flows or they too will trip off-line leading to a cascading outage. There is also te benefit of “interconnectedness” as your noted in the ant example, but in power systems that interconnected can provide help to a weakened part of the system (as we have seen with European transmission operators in ENTSO-E and extending that helps to Estonia in your example, but also Ukraine by connecting it to the ENTSO-E system at the outbreak of the Russian invasion.
So, with that being said, it is not the question of “optimization”, but the question of what constraints and contingencies are we accounting for when optimizing. I say this as a computational geek. You are calling the resilience, but I would refer to it as contingencies. You are calling it “slack”, and I would call it “reserves”. If we would only reframe the discussion, then the idea of optimization can actually help else think about how to be more resilient and handle unexpected events or opportunities.
The problem with the “Church of Optimization” is that its priests live in a world that is static and assumed to never change, and world where it is always to possible to operate at the ragged edge because change and contingencies do not happen, or they are simply ignored. Why do people ignore this? They cannot wrap their minds around uncertainty and ambiguity so they simply ignore because it is “too hard.”
Finally, in applying this to our personal lives it will only lead to misery and missing out on the opportunities and magic that may come and will be missed because we are so fixated with optimizing and checking boxes. BTW, checking boxes kills any kind of creative spirit and thinking that allows great discoveries to be made by allowing pour minds to wander and putting together seemingly disparate pieces of information that have crucial linkages.
Thank you, Brian. An interesting and important topic, very well written.
Can we reconcile the incessant raucous and social pressure to optimize all systems great and small with our quietly personal needs-and-wants that are begging for more resiliency, for less stress and strain? I’d like to think so. But it would seem to require a significant degree of personal “checking out” of these systems and of going “off-grid.” Eschewing social conformity and lessening our exposure to groupthink. At least every now and then. Preceded, necessarily, by taking the time and attention for an inward journey of discovering and knowing and accepting ourselves. Which is easier said than done. We’re much more capable of checking out and going off-grid when we’re emotionally well-centered and balanced, know who we are and what we need to be healthy and whole. When we no longer crave, or even much need, the external validations and affirmations offered by conformity with an unrelenting and unsparing optimization culture…
Great article! I worked for one year (that’s all I could stand) as Director of Data Center Efficiency at a large power company. We optimized our network by optimizing PUE or Power Usage Effectiveness, a metric that measures how efficiently a data center uses power. We had incredible PUE numbers and then one day a circuit breaker tripped in the middle of nowhere and the dominoes fell. Our numbers went way up because we were using no power at all. Neither were 37,000 of our customers. We were back on line in a few hours but I decided to apply my skills elsewhere.
"Former colleagues of mine subscribed to this list might recall that I was absolute dynamite during financial crises - no false modesty here, there were not many who could match me. They will also recall that outside of crises, I was kind of abject, like to the extent of regularly being unable to complete a simple Excel model. Everyone who employed me (and I was very lucky to find enough of them who understood this) had to accept that I was going to spend stretches of years at a time being a useless drag on the franchise, then make them a ton of money in years four and five."
One former coach described this as "hoplite" behaviour- which puzzled me. He explained that an Athenian hoplite would be a farmer, shop keeper , gardener etc, and would work for years at this. But at times of war and crisis , he would take out his armour from his shed, become a citizen hoplite and fight through the time of tension. Perhaps this has parallels with your expertise and experience?
There is another hazard in optimization that I've been thinking about for awhile and that is Entrainment. I believe there is a social science term for entrainment but I'm not sure what it is.
Entrainment is an interesting concept. I think it’s origin is meteorology and it describes the phenomenon of the natural flow of air being partially caught up, or entrained, seamlessly into another lifting flow without noticeable resistance. Entrainment is also used to describe something that pulls you in so naturally that you are unaware of the change. It is the unconscious tapping of your foot to the background music. Entrainment captures things at their natural frequency and flow.
While entrainment seems reasonably benign, it is what turns a Spring day into an angry thunderstorm that, without resistance, builds until it becomes a destructive force. It can be a force so powerful that it can be disrupted only when the atmosphere can no longer support its mass.
Regarding this discussion, Entrainment is the optimization that attracts resources so naturally that it scales to a size where it has subsumed most alternatives completely and competition or innovation cannot attract enough capital or energy to disrupt the flow. It is the sovereignty, the hegemony, the tyranny of the status quo. It is the cow path unconsciously followed that becomes the cart path that becomes the road that becomes the four lane and as such becomes entrenched as the status quo when on thoughtful reflection it could have, at any stage, been much better designed.
Our current chaotic capital allocation system (money follows money) does not seem to address this issue and I think we lose the resilience that reasonable allocation would instill at our peril.
NB: the Cybersecurity and Infrastructure Security Agency of the US government emphasizes resiliency in (at least) cybersecurity, communications, electric power delivery, and other critical infrastructure. They even provide programs for resilience planning and stress testing. This link goes to the search results (almost 24k) for the term resilience at cisa.gov. We tend not to trust the government because we have been trained not to do so by decades of media and political training (distraction) so this agency may not be able to undo that training, but there it is.
Great article. It brings to mind the global computer outage caused by American cybersecurity company CrowdStrike in July last year is a classic example of optimization for absolute efficiency having catastrophic consequences. Readers will recall the disruption at airports, airlines, banks, and medical centres headlining on the 19th July 2024 causing financial damage estimated at US$10 billion. The outage was caused by a defective Crowdstrike update distributed to all their customers using Windows, simultaneously. To do it’s cybersecurity thing Crowdstrike has to distribute updates (cyber threat counter measures) frequently, sometimes more than once a day, as a consequence testing the updates was absolutely efficient - it was automated. Flaws in the automated testing process failed to reveal the defect in the update that caused the outage…oops. These flaws have been rectified, but more importantly in my view, the hitherto simultaneous distribution has been updated “…with additional deployment layers and acceptance checks… designed to identify and mitigate potential issues before wider deployment” - which I take to mean Crowdstrike will, amongst other things, internally boot Windows machines after deploying updates on them to eyeball if bad things happen before releasing an update – not quite diversity and redundancy but it has parallels. As pervasive as software is in our world, it’s revealing that a large ship stuck in a small canal can cause seven times as much $ damage.
I still remember the impact reading “The Protestant Ethic” in college had on me. “Oh shit!” I thought, even though I wasn’t religious and had no concerns about being saved. “He’s got my number; this productivity bug is going to run my life.” At 84 I still feel guilty if I haven’t checked some boxes on my bottomless list.
Oliver Burkeman’s latest book is a perfect antidote to the optimization dilemma. I’m on my second time reading it.
Ammonites seem to provide a striking example of this.
After their first crude appearance as a simple swirl of a smooth-sided shell, with simple segmentation within, they evolved into a plethora of shapes, sizes. Even how the partition of the internal segments connected to the outer shell becoming a flourish of artistic variations on seemingly all possible patterns such a simple bond could be. Some abandoned the tight coil entirely, and grew more into an open, spiraling form. Like an artist discarding convention and expanding on a more expressionist concept of their body-plan.
Then, in the thundering FADOOOOOM, characteristic of a mountain landing at 27,000mph, they where all gone.
All that is, except for the humble Nautilus, with that most basic, yet evidently most resilient design.
An excellent article. I took my elderly, and not very mobile , mother to Kings College Cambridge recently. I was in check list mode- see Kings ( tick); see Trinity (tick); etc. But my mother suggested we just sit down and look at the Kings College court. And observe. And I got a lot more out of the experience.
Great piece! We have experienced this firsthand in medicine….my oncologist husband and I (ob-gyn) as well as children in healthcare live this daily…there is a point of diminishing return in terms of quality of care and ability to be resilient for cases that don’t fit the algorithm. The health system we were part of LIVED this to the extent that an entire hospital was built on their model of efficiency…now dismantled. They had an entire department devoted to their Improvment systems….We wasted weeks in meetings, events, post it notes on the wall with the group think. One delivery model was instituted over 3 years..the staff-physicians, nurses, everyone told them it was a bad idea…but we forged on, now dismantled as well. There was no open door policy to discuss with leadership…they had paid high price consultants to tell them what to do….and so it was….a sad day for the providers and patients. How can you fight this? My husband’s group was independent but sadly market forces and the hospitals desire to capitalize on the chemotherapy profits put them squarely out of business. The “you play in my sandbox or go home” is also part of this. I am certain other industries have experienced this but it is acutely an issue in medicine.
Excellent article. In my career in IT, both programming and as an analyst, we were called on to solve problems. At a large insurance company that I worked for at the time, they were going to measure and improve efficiency on all processes. I argued with my manager that maybe that worked for issuing quotes, but they couldn’t measure my “thought work” and creativity. He said I was wrong. In the end, the company abandoned the measurement initiative, at least in IT. I’ve been through so many versions of this optimization idea, it’s laughable. I learned way too late in life (I’m now semi-retired and work as an aide for my special needs adult child) the lessons from the Reddit thread you posted. But I’m enjoying life so much more now without checklists. Sadly, I believe your message will go unheeded by those in power, at least until it’s too late and the systems break.
I wrote a couple of papers making similar points but never got them published.
More interestingly, and totally by chance (!), I ran across a copy of Fluke at my local library and read it in one sitting. Great stuff! I'll try to write some comments soon
Our current culture to strive needs to reflect on the importance of sauntering through life.
Quite agree, Ken.
"...despair and the incapacity for leisure are twins..."
"...a certain happiness in leisure...the happiness that comes from the recognition of the mysteriousness of the universe and the recognition of our incapacity to understand it...so that we are content to let things take their course....which Konrad Weiss, the poet, called'confidence in the fragmentariness of life and history'..."
-Josef Pieper, Leisure: The Basis of Culture ( pp. 26-27)
and Bertrand Russell:
"" Leisure is essential to civilization, and in former times lesiure for the few was only rendered possible by the labours of the many. But theiur labours were valuable, not because work is good, but because leisure is good"
- In Praise of Idleness
This is excellent. Very inspiring to, so here's a few thoughts/add ons.
1) Notions of resilience make me always think about the Black Death which killed 30, possibly up to 60% (!!!!) of the population of some (most?) parts of mediaeval Europe within a relatively short period. And while it was a cataclysmic event which caused much social upheaval, what I think is remarkable yet not enough remarked on is that *it did not lead to a collapse of the civilisation present in Western Europe at the time*. One in three (or more) people died horrible and scary deaths yet the rest, somehow, went on. And while I know that whole communities descending into "animal barbarity" in times of disaster is to a large extent a myth, the resilience here, not just of individuals or small kin groups but institutions and structures was remarkable.
Obviously some of it was due to a relatively low complexity and centralisation of the "civilization" then. But still.
2) During the few times when I was employed in vastly differing roles, I repeatedly experienced the staffing version of just-in-time "agile" processes: optimising headcounts meant that while everything ticked over fine in normal times, any unpredicted extra work or anyone taking time off for whatever reason meant the organisation or team began struggling. And that often could not be resolved by return to normal, because the normal was optimised for normal. Lack of labour redundancy meant that it was very hard to make up.
3) In addition to sapping joy out of life, it feels like over optimising will reduce creativity and innovation, because one needs the mental spaces of in-between freedom to allow for the hatching and cooking of new ideas.
4) All that said, I feel that the message of "don't live over optimised checklist life" risk something similar to "it's ok to prioritise yourself" message of current pop psychology: the people who will pick it up will be people who really should make up a checklist or two while the ones to whom it's relevant will often add an item of "build in slack into my pomodoro schedule" to their to-do list. I'm saying it as a lazy wanderer who only recently (in my dotage, pretty much) acknowledged that SOME routine and goals are valuable.
"it did not lead to a collapse of the civilisation present in Western Europe at the time" is an interesting point. While obviously there continued to be people and houses and farms and so on, I've seen people suggest that the dramatic increase in power by the surviving labourers indirectly led to economic and political changes that would ultimately give us the end of feudalism and absolute monarchy centuries later. So while society as a whole was resilient, the political system based on serfdom was much more brittle.
Very good point. The system "broke," but not in a way that led to a general collapse. 🤔
Brian, great piece! As somebody who works in power system economics and operation the Estonian case is typical. Power systems are “optimized” subject to constraints and contingencies such as the loss of a large generator or transmission line. Transmission facilities (lines and transformers) are not fully loaded up to account for these contingencies as if we lose a generator or line, the topology has changed and power flows will change and there must be room left on other lines to handle additional flows or they too will trip off-line leading to a cascading outage. There is also te benefit of “interconnectedness” as your noted in the ant example, but in power systems that interconnected can provide help to a weakened part of the system (as we have seen with European transmission operators in ENTSO-E and extending that helps to Estonia in your example, but also Ukraine by connecting it to the ENTSO-E system at the outbreak of the Russian invasion.
So, with that being said, it is not the question of “optimization”, but the question of what constraints and contingencies are we accounting for when optimizing. I say this as a computational geek. You are calling the resilience, but I would refer to it as contingencies. You are calling it “slack”, and I would call it “reserves”. If we would only reframe the discussion, then the idea of optimization can actually help else think about how to be more resilient and handle unexpected events or opportunities.
The problem with the “Church of Optimization” is that its priests live in a world that is static and assumed to never change, and world where it is always to possible to operate at the ragged edge because change and contingencies do not happen, or they are simply ignored. Why do people ignore this? They cannot wrap their minds around uncertainty and ambiguity so they simply ignore because it is “too hard.”
Finally, in applying this to our personal lives it will only lead to misery and missing out on the opportunities and magic that may come and will be missed because we are so fixated with optimizing and checking boxes. BTW, checking boxes kills any kind of creative spirit and thinking that allows great discoveries to be made by allowing pour minds to wander and putting together seemingly disparate pieces of information that have crucial linkages.
Thank you, Brian. An interesting and important topic, very well written.
Can we reconcile the incessant raucous and social pressure to optimize all systems great and small with our quietly personal needs-and-wants that are begging for more resiliency, for less stress and strain? I’d like to think so. But it would seem to require a significant degree of personal “checking out” of these systems and of going “off-grid.” Eschewing social conformity and lessening our exposure to groupthink. At least every now and then. Preceded, necessarily, by taking the time and attention for an inward journey of discovering and knowing and accepting ourselves. Which is easier said than done. We’re much more capable of checking out and going off-grid when we’re emotionally well-centered and balanced, know who we are and what we need to be healthy and whole. When we no longer crave, or even much need, the external validations and affirmations offered by conformity with an unrelenting and unsparing optimization culture…
Great article! I worked for one year (that’s all I could stand) as Director of Data Center Efficiency at a large power company. We optimized our network by optimizing PUE or Power Usage Effectiveness, a metric that measures how efficiently a data center uses power. We had incredible PUE numbers and then one day a circuit breaker tripped in the middle of nowhere and the dominoes fell. Our numbers went way up because we were using no power at all. Neither were 37,000 of our customers. We were back on line in a few hours but I decided to apply my skills elsewhere.
Dan Davies had something to say about this recently: https://backofmind.substack.com/p/a-more-subtle-cost-disease
"Former colleagues of mine subscribed to this list might recall that I was absolute dynamite during financial crises - no false modesty here, there were not many who could match me. They will also recall that outside of crises, I was kind of abject, like to the extent of regularly being unable to complete a simple Excel model. Everyone who employed me (and I was very lucky to find enough of them who understood this) had to accept that I was going to spend stretches of years at a time being a useless drag on the franchise, then make them a ton of money in years four and five."
One former coach described this as "hoplite" behaviour- which puzzled me. He explained that an Athenian hoplite would be a farmer, shop keeper , gardener etc, and would work for years at this. But at times of war and crisis , he would take out his armour from his shed, become a citizen hoplite and fight through the time of tension. Perhaps this has parallels with your expertise and experience?
There is another hazard in optimization that I've been thinking about for awhile and that is Entrainment. I believe there is a social science term for entrainment but I'm not sure what it is.
Entrainment is an interesting concept. I think it’s origin is meteorology and it describes the phenomenon of the natural flow of air being partially caught up, or entrained, seamlessly into another lifting flow without noticeable resistance. Entrainment is also used to describe something that pulls you in so naturally that you are unaware of the change. It is the unconscious tapping of your foot to the background music. Entrainment captures things at their natural frequency and flow.
While entrainment seems reasonably benign, it is what turns a Spring day into an angry thunderstorm that, without resistance, builds until it becomes a destructive force. It can be a force so powerful that it can be disrupted only when the atmosphere can no longer support its mass.
Regarding this discussion, Entrainment is the optimization that attracts resources so naturally that it scales to a size where it has subsumed most alternatives completely and competition or innovation cannot attract enough capital or energy to disrupt the flow. It is the sovereignty, the hegemony, the tyranny of the status quo. It is the cow path unconsciously followed that becomes the cart path that becomes the road that becomes the four lane and as such becomes entrenched as the status quo when on thoughtful reflection it could have, at any stage, been much better designed.
Our current chaotic capital allocation system (money follows money) does not seem to address this issue and I think we lose the resilience that reasonable allocation would instill at our peril.
NB: the Cybersecurity and Infrastructure Security Agency of the US government emphasizes resiliency in (at least) cybersecurity, communications, electric power delivery, and other critical infrastructure. They even provide programs for resilience planning and stress testing. This link goes to the search results (almost 24k) for the term resilience at cisa.gov. We tend not to trust the government because we have been trained not to do so by decades of media and political training (distraction) so this agency may not be able to undo that training, but there it is.
https://www.cisa.gov/search?g=resilience+cisa
Great article. It brings to mind the global computer outage caused by American cybersecurity company CrowdStrike in July last year is a classic example of optimization for absolute efficiency having catastrophic consequences. Readers will recall the disruption at airports, airlines, banks, and medical centres headlining on the 19th July 2024 causing financial damage estimated at US$10 billion. The outage was caused by a defective Crowdstrike update distributed to all their customers using Windows, simultaneously. To do it’s cybersecurity thing Crowdstrike has to distribute updates (cyber threat counter measures) frequently, sometimes more than once a day, as a consequence testing the updates was absolutely efficient - it was automated. Flaws in the automated testing process failed to reveal the defect in the update that caused the outage…oops. These flaws have been rectified, but more importantly in my view, the hitherto simultaneous distribution has been updated “…with additional deployment layers and acceptance checks… designed to identify and mitigate potential issues before wider deployment” - which I take to mean Crowdstrike will, amongst other things, internally boot Windows machines after deploying updates on them to eyeball if bad things happen before releasing an update – not quite diversity and redundancy but it has parallels. As pervasive as software is in our world, it’s revealing that a large ship stuck in a small canal can cause seven times as much $ damage.
In the 1960s it was called building in a contingency factor aka common sense.
I still remember the impact reading “The Protestant Ethic” in college had on me. “Oh shit!” I thought, even though I wasn’t religious and had no concerns about being saved. “He’s got my number; this productivity bug is going to run my life.” At 84 I still feel guilty if I haven’t checked some boxes on my bottomless list.
Oliver Burkeman’s latest book is a perfect antidote to the optimization dilemma. I’m on my second time reading it.
Ammonites seem to provide a striking example of this.
After their first crude appearance as a simple swirl of a smooth-sided shell, with simple segmentation within, they evolved into a plethora of shapes, sizes. Even how the partition of the internal segments connected to the outer shell becoming a flourish of artistic variations on seemingly all possible patterns such a simple bond could be. Some abandoned the tight coil entirely, and grew more into an open, spiraling form. Like an artist discarding convention and expanding on a more expressionist concept of their body-plan.
Then, in the thundering FADOOOOOM, characteristic of a mountain landing at 27,000mph, they where all gone.
All that is, except for the humble Nautilus, with that most basic, yet evidently most resilient design.