A Systematic Approach to Experimenting with Gen AI

Extractive Summary of the Article | Read | Listen

After taking the software industry by storm, generative AI is now moving into a broad set of industries, including manufacturing, where it is helping manage unpredictability and support real-time decision-making. Gen AI’s ability to codify, automate, and distribute organizational expertise may eventually reshape work structures from the shop floor to the C-suite. Already some companies are using it to analyze the flood of information generated in factories and to predict problems, simulate complex scenarios, and optimize processes in real time. By working with a wide range of manufacturing industry data—from maintenance manuals and machine automation code to complex diagrams, 3D drawings, and process data—gen AI has the potential to establish new ways for people and machines to collaborate.

But who will benefit from these changes, and how quickly? That’s not a simple question. Like electricity and the printing press, gen AI is a general-purpose technology—the adoption of which, history teaches us, is rarely straightforward. Managers often fail to recognize the true economic potential of new technologies and struggle to reorganize tasks, skills, and workflows to suit them. As a result, performance gains typically lag behind technological diffusion, giving rise to what’s known as the “productivity J curve”: an initial dip in productivity as organizations adapt to a new technology, followed by sustained gains once complementary investments pay off.

Because it’s not clear how firms will adopt gen AI, managers face a strategic dilemma: Wait for more clarity and risk falling behind? Or act too soon and invest in applications that don’t deliver?

To address this tension, leaders need to think about gen AI adoption not as a single decision but as a portfolio of organizational experiments. Like A/B testing in digital-product development, these experiments should aim to isolate causal effects—focusing not just on whether gen AI works but also on how it works, for whom, and under what conditions. By testing gen AI applications before scaling them up, managers can reduce risk, refine their strategies, and build internal momentum for change. Experts have been advocating for this approach, but many firms are struggling to implement it. Experimentation therefore remains a relatively novel practice in many organizations.

That needs to change. Experimentation allows companies to transform gen AI uncertainty into a strategic advantage. It helps firms move through their own adoption phase more successfully than their competitors do. And the knowledge generated through experimentation can be leveraged to reinforce existing relationships—or create new ones—within their ecosystems. In this article the author describe how firms are getting better at adopting gen AI through experimentation—within their own organizations and across entire ecosystems.

Engines of Learning and Adaptation.  At its core an organizational experiment is an application of the scientific method. In a real-work setting, it establishes a treatment group (for example, employees or teams using a new AI system) and a control group (those operating as usual, without the new system).  When randomization is difficult, some firms implement staggered rollouts, phasing an intervention in to different teams over time to create natural control groups.  Another approach is to create a “lab in the field”—that is, a controlled environment where interactions with the new technology can be observed.

When conducted properly, organizational gen-AI experiments can produce a host of benefits, including the following:  Causal insights, granularity, risk reduction, and strategic learning.  

Ecosystem Experimentation.  Gen AI experimentation does not simply benefit potential adopters. Innovators can see even greater returns: They can apply the insights gleaned to help prospective buyers understand which gen AI use cases really matter for them or which challenges might prevent them from integrating the technology into existing processes. Some innovators with large user bases are leading the experimentation surrounding new gen AI applications outside their own organizations—for example, in partnership with current or prospective buyers. In these cases, innovators orchestrate ecosystem experimentation.

To become an organizational experimenter successfully, you’ll need to focus on several critical areas:  Customer needs.  Usable prototypes.  A learning mindset.  Experimental expertise.  And partnership capabilities.

show less