A pilot speaks into the intercom during the flight, saying, “I have some good news and some bad news. The bad news is that we are lost; the good news is that we are progressing very well.”
I hope this story is funny enough for you because when it comes to businesses, the story would not be funny anymore, especially since many organizations are in that kind of cockpit. The dashboards show green and the KPIs are rising. Productivity per head has never been higher. Yet no one has taken the time to consider whether the plane is heading for a destination worth landing at. I remember reading somewhere somebody saying: “Being quick has nothing to do with direction, and efficiency is not the same as purpose.”
Since reading an article about artificial intelligence in Aeon,[1] I have been considering this point. The author explains that he heard a well-respected computer scientist say that AI systems optimize, that they become better at optimizing through experience, and that, as he put it, this is what constitutes intelligence. The audience responded by nodding. The writer himself did not. Why is that so? Because optimization always requires a target, and the machine doesn’t select the target; someone else does.
A navigation program determines the quickest way to a place it never chose itself. A chess engine plays to win a game it was never asked to play. That is excellent engineering. But does that constitute all there is to intelligence? The author says this old argument is worth revisiting.
In the eighteenth century, David Hume, the Scottish philosopher, famously said that reason is “the slave of the passions,” meaning our desires choose the destination while reason only calculates the route. A few decades later, Immanuel Kant objected. He maintained that reason does more than discover efficient means; it can step back and consider whether an end is worthy from the start. Can it be justified? Does it regard people as persons or as tools? Hume seems to provide a clever servant, and Kant a responsible leader.
Many companies may operate according to Hume’s model without realizing it. They excel at optimization: setting up cascading targets, monitoring metrics, and rewarding those who hit the figure. But they seldom ask whether the figure merits being achieved. Economists refer to one form of Goodhart’s law as stating that when a measure is turned into a target, it is no longer a good measure.
If a call center is rewarded for making short calls, then the agents will begin to cut customers off. If a sales team is rewarded only on volume, someone will eventually sell the client something he or she does not need. The target is reached, but the original purpose is undermined. A metric should be regarded as a servant and must never be allowed to take on the role of a master.
If an organization stops assessing its objectives, those objectives do not vanish. Instead, they become fixed defaults: growth, speed, engagement and profit. No one deliberately selected them; they have just become the standard features. And since no one has chosen them, no one questions them.
The algorithm makes recommendations. The dashboard ranks priorities. The system forecasts outcomes. The model summarizes the data and tells us what deserves attention. And there is tremendous convenience in all of this. AI can sift through thousands of data points, detect patterns we might miss, identify risks, compare alternatives and surface options in seconds. That is valuable.
But a subtle danger also exists. The more capable our tools become, the easier it becomes to surrender not only the work of analysis but also the responsibility of judgment. There is an important difference between asking AI, “What is happening?” and asking it, “What should matter?” The first is largely analytical. The second is profoundly human.
AI can identify profitable customers, rank employee performance, recommend cost reductions and predict financial returns. But it cannot determine which customers deserve our loyalty, what kind of people we should develop or whether our decisions reflect our values. These are not merely calculations. They are leadership judgments.
Data tells us what is. Predictions suggest what may happen. Optimization identifies what is efficient. But leadership must determine what is right, what matters and what is worth pursuing. Technology can recommend decisions, but it cannot assume accountability for their consequences.
Good leaders will distinguish themselves from AI through judgment, courage, empathy and wisdom.
In every strategy meeting, ask:
1. Is the goal worth pursuing?
2. What behaviors will our KPIs encourage?
3. Who might our success harm?
Use AI to improve efficiency but never surrender leadership responsibility. Let AI improve the how. Let data inform the what. But leadership must own the why.
Catch Kongversations with Francis on YouTube and all major podcast platforms: Spotify, Apple Podcasts, Google Podcasts and more. Plus, listen to Inspiring Excellence wherever you stream.