[This is the second part of a three-part series. To read more, see Part 1 here.]

ChatGPT’s launch nearly three years ago brutally revealed to the entire world — from business leaders to struggling high school students — the astonishing capabilities of AI. It sent a clear message to the business world. But rather than providing answers, it raised a series of existential questions. “This is a revolution, but what kind of revolution is it?” Even while speculating about the highly uncertain answer to that question, a deeper one became apparent: “What’s my role in it?”

In part 1 of this three-part series, Vinay Singh highlighted the fact that organizations are failing to understand the areas in which they need to focus their investments. They tend to rely on their old habits and instincts, born in an era when AI was simply some distant laboratory’s futuristic research project. 

But today, something radical is taking place that goes beyond traditional organizational strategy. The AI challenge has everyone wondering about what emergence means for the future of humanity. For traditional industries (i.e., any business that was not selling AI) and for professionals in nearly every sector, time-tested methods are no longer adequate. It has become urgent to ask a focused question: “What impact will the AI revolution have on my business?”

A Massachusetts Institute of Technology (MIT) study group has focused precisely on that question and released the results in a 26-page document. The media wasted no time summarizing the report’s findings by citing its most salient observation: “Just 5% of integrated AI pilots are extracting millions in value, while the vast majority remain stuck with no measurable [profit and loss] P&L impact.”

We should all recognize that the media does two things to orient the public’s perception of whatever new topic they report on. The first is to simplify complex questions for easy consumption. The second is to exaggerate the significance of what they have already simplified. The authors of the MIT study have provided the perfect occasion for the media to do just that by announcing their statistic of a 95% failure rate. A closer examination of the report’s content reveals a much more complex situation.

The rather shocking conclusion implied by the 95% failure rate reflects a lazy philosophical position most casual readers have been conditioned to share. It consists of accepting the premise that, in matters of business, P&L is the ultimate metric and the most reliable indicator of truth. 

While the photograph of any year’s P&L does tell an important story, anyone seriously interested in uncovering the truth about the impact on performance of an emerging form of technological or organizational innovation should take that metric seriously but look beyond it. The result will always be more nuanced than what a single indicator can reveal. Although it may sound heretical, sometimes anecdotal evidence can seriously modify the perception produced by a single statistic.

The “shadow economy”… curiously left in the shadows

At one point, the study’s highlights reach beyond traditional reasoning when they consider what the study calls “the shadow AI economy.” It even sees this as a potential “bridge across the divide,” noting that “forward-thinking organizations are beginning to bridge this gap by learning from shadow usage and analyzing which personal tools deliver value before procuring enterprise alternatives.” Oddly, the study treats this as a marginal phenomenon that literally “throws no shadow” on its pessimistic conclusion about the failure to affect P&L.

This is precisely where anecdotal evidence could be helpful. If, for example, the use of AI has had a positive effect on an operative’s self-belief (such as, “now I don’t have to anguish over a blank page when drafting a report”), even though it produces no immediately measurable result, this could be an indicator of something profound and lasting that will play out over time. Based on traditional performance indicators, however, management will typically write this potential breakthrough off because its impact on the bottom line cannot be measured and calculated.

Furthermore, the psychological effect of transformative tools in the workplace may begin to have an effect on each individual, but ultimately, similar experiences will produce a collective result. Typically, personal productivity will go down in the early phases of adoption because of operational and cultural uncertainty. Measuring the effect on productivity too soon, before appropriate adjustments can be made, will almost certainly lead to the cynical conclusion that it’s all hype.

For those who read beyond the headlines, the study does awkwardly acknowledge this dimension of the question in its consideration of the shadow economy. It notes that “these tools primarily enhance individual productivity, not P&L performance.” This absurdly implies that individual productivity is unrelated to P&L. That may be true in the short term, but ignoring the fact that the effects of innovation take place over time reflects a form of organizational blindness. 

In its quest to be “objective” and scientific, this kind of management fails to acknowledge the time factor. In any organizational context, everyone’s individual experience will gradually and naturally define the patterns, habits and reflexes of the professional culture. When technology changes, culture — first professional culture, then corporate culture — inevitably follows suit.

Seeking a new conception of performance indicators

The real question to ask in all such situations is not what the traditional indicators tell us, but rather whether we, as organizational decision-makers, possess the tools and intellectual flexibility to recognize and manage the processes and effects of inevitable behavioral change. It’s either that or risking the loss of control.

The study goes into some depth to assess the causes for disappointment. “Most fail due to brittle workflows, lack of contextual learning, and misalignment with day-to-day operations.”  Could it be that those very “day-to-day operations” need to evolve? The Darwinian law of survival of the fittest implies that genetic structures adapt to changing environmental conditions to ensure survival. “Brittle workflows” can be made more robust. 

All serious learning, the key to improved productivity, is contextual, rooted in the reality with which it interacts. Rather than questioning AI’s ability to adapt to the no longer optimal work habits inherited from the past, it makes more sense to engage in contextualized learning, something that AI, accompanied by human ingenuity and flexible management skills, has the capacity to implement.

One reading of this analysis of failure is that the disappointing effects highlighted in the study reveal gaps in human managerial skills in the areas of time management, full-spectrum perception and the capacity to deal with complexity in an evolving context. Our technology and its rate of evolution have cast all of us into a VUCA world where everything at some point can become Volatile, Uncertain, Complex and Ambiguous. The failure acknowledged by the study indicates the need to rethink the system and focus on each of its factors. The very term “failure” reflects the kind of absolutist judgment that often prevents success from ever being achieved.

In its detail, the MIT study helpfully identifies some of the specific symptoms of this pattern of failure.

• Limited disruption: Only two of eight major sectors show meaningful structural change.
• Enterprise paradox: Big firms lead in pilot volume but lag in scale-up.
• Investment bias: Budgets favor visible, top-line functions over high-ROI back office.
• Implementation advantage: External partnerships see twice the success rate of internal builds.

This reflects the lack of preparation in current business operations models. One obvious feature is the reflex built into our business and general culture of “short-termism.” The fact that the study highlights the success of external partnerships is an example of the temptation to focus on the short term. This recommendation definitely makes sense today, but may be more questionable over time. 

There is nothing wrong with the strategy of seeking external partners who have not only dealt with, but also solved issues that the struggling enterprise in question has not anticipated and prepared for. But rather than seeing this as “the solution,” a wiser approach would be to look at it as a transitional strategy, a learning moment. This highlights the endemic problem of failing to distinguish between time frames — short, medium and long term — as well as solely relying on traditional metrics.

The dynamics of learning and evolving corporate culture

Interestingly, the study at one point does acknowledge what we should understand as possibly the most essential factor: evolution through the process of learning. Learning, by definition, is a dynamic, open-ended process. The study, however, appears to frame the question in static terms, referring to the learning question as a problem rather than an opportunity. “The core barrier to scaling is not infrastructure, regulation, or talent. It is learning.” So far, so good. But then we are told: “Most GenAI systems do not retain feedback, adapt to context, or improve over time.”

Although factually founded as a description of existing systems, this observation fails to appreciate the obvious fact that technology itself is not static, quite the opposite. It is evolving and, moreover, with a bit of coaxing, it can even be made to evolve. Learning is indeed the key. But we need to take on board the fact that learning is more than the acquisition of static knowledge.

Multiple types of learning exist, and this is true of both machines and humans. We are currently entering an era in which the interaction between humans and machines will have a transformational effect on all productive activity. Because AI “thinks” and produces content that we deem exploitable, and because it has a voice that can draw conclusions and either make decisions or influence human decisions, the way businesses and indeed all professional activities are managed will inevitably become more complex. For both humans and machines, this will be a complex learning process. It’s also one they can now work on together, if, that is, they learn how to collaborate.

As a starter, if we want to explore what it means when we delve into the deeper question of learning on the human side, we need to identify the different levels of acquisition and understanding achieved while learning. It’s important to distinguish three key goals:

• Developing strategic understanding.
• Improving management skills.
• Refining operational skills.

Ultimately, the three combine to describe an evolving enterprise system. In parallel, our human skills and AI tools will evolve as they interact over time. All three of these areas of dynamic improvement (for AI, managers and operators) require not just human oversight but human accountability. 

The MIT study may simply be wrong when it states, “Most GenAI systems do not retain feedback, adapt to context, or improve over time.” This is a true description of the level of performance of “most of” today’s tools, but it denies the fundamental reality of today’s workplace. For the first time, we have tools that are designed to self-improve. It’s time that we invent human strategies that also include at their core the idea of self-improving, or “deep learning.”

The path forward implies a serious effort at achieving the redesign of the way our businesses operate. AI is one factor whose dynamic growth will pose a permanent challenge. Geopolitics is emerging as another one — a question we will come back to in a different context. The simple truth is that we can no longer feel comfortable with the “rules-based” management order that has prevailed during the first phase of the digital age. All businesses need to rethink their approach to strategy, management and operations now that AI will inevitably be not only a part of the conversation, but also a contributor to it. This also means rethinking our approach to performance assessment, whether we’re speaking of the performance of strategists, managers, operators or AI itself.

[Kaitlyn Diana edited this piece.]

The views expressed in this article are the author’s own and do not necessarily reflect Fair Observer’s editorial policy.