Executives in every large organization were convinced at the outset of 2023 that artificial intelligence (AI) and generative AI (gen AI) would have a big impact on how their companies operate. Organizations invested enormous resources into gen AI, expended a lot of energy thinking about how the technology could be best put to use, and initiated numerous pilots aligned to those use cases.
Most of those Proof of Concept (POC) pilots – approximately 90% – will not move into production in the near future, and some may never move into production. However, some companies are successfully moving some of their pilots into production.
Everest Group, in partnership with Yates Ltd. and CalypsoAI, convened a panel of 50 CIOs and CTOs in Fortune 500 and other large enterprises throughout last year for ongoing conversations. The group reflected on their gen AI journeys and sought and shared information about how to invest in and leverage this fast-moving technology wisely.
Findings: The Problems with Pilots
The list below presents snapshots of the five most significant findings regarding the gen AI pilot dilemma.
The Technology – Many companies pilot gen AI for use cases in which it is the wrong technology to address the business need. In some cases, the companies found better technologies, already developed or even already in place, that met their needs better than gen AI. In other cases, they found that gen AI is not yet mature enough to execute the tasks robustly in the production environment.
Change Management – Change management issues at the pilot level are formidable and take a lot of time and resources to work through. Yet, they must be resolved before the pilot can achieve success in a production environment.
Risks – Pilot programs frequently present unforeseen, and therefore unplanned for, but nonetheless significant risks and uncertainties associated with security and intellectual property.
Return On Investment – The return on investment (ROI) can be difficult to quantify after a pilot is put into production. Many of the benefits remain speculative at this point and hard to guarantee. Without deeper confidence about the expected ROI, companies are reluctant to move forward and spend significant amounts of money placing pilots into production.
Funding – In many cases, gen AI can be an expensive technology to both deploy and, more importantly, to run and maintain after it is operational in production.
Exacerbating these issues is the current budgetary situation in which large organizations must carefully control their technology budgets while stuck in a more-for-less phase of tech spending – effectively robbing Peter to pay Paul. This mindset causes companies to prioritize options rather than fund a lot of options and, when added to the five problems listed above, prevents most pilots from moving into production.
The combination of these hurdles to successfully evolving gen AI tools from pilots to production can be summed up in one concerning term: pilot fatigue. Companies are simply weary of spending more time, money and energy to support pilots that do not progress into production quickly or at all.
Where Are the Viable Opportunities?
Despite the pilot fatigue, executives on our panel discussed their still-strong belief that there are great opportunities for gen AI. They remain convinced that, where opportunities exist, tools that move from pilots into production could be very powerful, beneficial technology solutions for their companies.
One of the convictions that the CIOs and CTOs recognize is that, while perhaps only 5-10% of the use cases under experimentation could bring opportunities, those could have a huge impact on their company and are worth pursuing.
Pilots can also identify additional use cases not initially considered by the program designers, and that subset of unforeseen use cases can lead to sustainable value.
A Significant Complication
CIOs and CTOs increasingly realize that the key decision-makers on where to deploy gen AI will be budget holders who have responsibility for business and functional organizations such as the head of sales, marketing, or supply chain. These executives tend to be risk-averse and highly practical, and they have little time for the intricacies of neural networks, IP violations, and the other myriad of technical details.
However, given their own experience with ChatGPT, they are eager to see where gen AI could benefit their business areas. Simply put, they do not want to discuss how gen AI works or where it could potentially be made to work. Instead, they want to know where it is working in companies in the areas for which they have responsibility. Specifically, they want to know information such as:
● In their area of responsibility, where is gen AI already in production?
● Which companies similar to the one where they work have gen AI in production?
● Which tools has that company deployed?
● What are the practical outcomes that the company has experienced?
Consequently, Everest Group’s panel of enterprise CEOs and CIOs asked us to give them information about where gen AI is working, for instance, within a specific business function. Armed with this information, they can then better focus their time and attention on high-return areas where their business units likely will have greater enthusiasm about funding new technology and working through the painful issues of change management, as well as managing the security and intellectual property risks.
We will post monthly tracking reports on our website, which will show by business function which gen AI pilots move into production, which tools are used, and what benefits are emerging in the deployment environments.
