Gartner - 10 Best Practices For Scaling GenAI
Gartner - 10 Best Practices For Scaling GenAI
the Enterprise
Published 10 January 2024 - ID G00804998 - 21 min read
By Analyst(s): Arun Chandrasekaran, Leinar Ramos, Alberto Pietrobon
Initiatives: Digital Future; Artificial Intelligence
Overview
Key Findings
■ Generative AI (GenAI) has the potential to transform businesses across industries.
Most business and technology leaders believe that the benefits of GenAI far
outweigh its risks, despite the significant risks it poses and potential regulations
likely to emerge in the near future.
■ Formulating a robust strategy for GenAI that aligns with the business strategy
continues to be a challenge for technology leaders. Responses to GenAI range from
banning it outright to ad hoc experimentation at an individual or department level.
Lack of understanding about emerging industry best practices is constraining
organizationwide pilots and scalable production deployments.
■ The pace of change in the GenAI ecosystem makes it hard for IT leaders to choose
the right approaches and technologies, as the ecosystem is constantly in a state of
flux, which exacerbates supplier selection and renders long-term technology plans
obsolete.
■ IT leaders still see major hurdles to adoption and scaling, such as a skills and talent
shortage, poor data quality, and lack of comprehensive AI governance, risk
mitigation and control.
■ Instill AI engineering practices that enable seamless scaling of your pilots such as
agile thinking, sandbox environments for rapid experimentation, composable
platform architecture, FinOps and a product-centric delivery model.
■ Invest in data and AI literacy skills with personalized training programs that focus on
the “doing” as much as the “knowing.” Maintain a balance between the individual
and organizational needs, giving employees the flexibility to customize their own
training journey.
Introduction
GenAI promises to be one of the most transformative technology trends of this decade.
GenAI can help CTOs tackle complex challenges, innovate at speed and build sustainable
competitive advantages in an ever-evolving digital economy. For CTOs and technology
innovation leaders, it represents not just an advancement in AI, but also a trend capable of
redefining operational efficiency, product development and customer engagement. GenAI
has changed the equation between boards and CEOs versus CIOs and CTOs, ushering a
tighter collaboration and collaborative learning on the art of the possible.
A Gartner webinar poll conducted in September 2023 reveals that more than three-quarters
of technology leaders are optimistic that the benefits of GenAI far outweigh its risks (see
Figure 1). 1
Given its strategic importance to the business, formulating best practices for scaling
GenAI is crucial for gaining stakeholder confidence, enabling innovation across the
organization, and building sustainable and adaptable AI strategies that can meet future
needs. While the GenAI ecosystem is rapidly evolving, we believe these emerging top
practices will remain evergreen and help in scaling GenAI in a way that maximizes
benefits, minimizes risks, and ensures that the technology continues to support and
enhance business objectives.
Analysis
The 10 best practices to scale generative AI are detailed below and summarized in Figure
2.
Importantly, the task of prioritizing should be a collective decision, involving not only the
IT and technology teams but also the business lines that will utilize the GenAI application:
■ Create a framework to measure and track business value, testing each use case at
the pilot stage and monitoring benefits after deployment. Ensure that business value
measures are specific, tangible and time-bound (see Assess the Value and Cost of
Generative AI With New Investment Criteria).
Ideally, you want to build when the AI product can give you a competitive differentiation in
your industry and when you have adequate skills and know-how for the build process. In
the context of GenAI, use cases where you want to minimize risks for regulatory or brand
equity reasons may also warrant a build approach. However, for most core business
functions (such as HR, supply chain, marketing, sales and IT) — either a GenAI embedded
application vendor or a GenAI native solution might be a more feasible approach.
There are a variety of approaches to steer and customize GenAI models — such as in-
context learning, retrieval augmented generation (RAG) or fine-tuning. Use the guidance in
How to Choose an Approach for Deploying Generative AI to analyze the pros and cons of
these approaches for your use case. In addition:
■ Evaluate the model training process, security and privacy practices, depth of
integration, ease of prompt engineering, and pricing model of your software vendor
when deciding to buy GenAI applications.
■ Categorize the use cases that you identified earlier across these various approaches.
Revisit this frequently as more efficient and even hybrid techniques (RAG plus fine-
tuning) emerge in the market.
To ensure the success of your pilot projects, obtain executive sponsorship, set the right
expectations of project outcomes, focus on business value early on and reduce technical
debt ahead of the pilot:
■ Adopt an agile mindset and start experimenting and testing these use cases to
determine the next step — scale, refine or stop. Don’t be thwarted by early failures —
figure out why the use case failed and then determine whether to refine or stop. Use
the pilot to refine your assumptions on the cost and value of scaling each use case.
■ Create a multidisciplinary tiger team to prototype and test these use cases
combining expertise in data science, IT, security and business. The success of GenAI
initiatives strongly hinges on data quality, models chosen, implementation approach
and ability to iterate.
■ Set up a sandbox environment for safe experimentation across the organization with
adequate security and privacy controls and make multiple GenAI models available to
experiment and iterate within the sandbox. This provides developers the flexibility of
choosing the right models for the right use case.
■ Determine the right model fit for your use case based on model performance, cost of
ownership, and security and privacy principles.
■ Invest in AI engineering tools for data integration, data serving, automated model
deployment, application development, model monitoring and responsible AI, where
these tools are agnostic to the underlying models but are tightly integrated with
them.
■ Define and publicize a vision for responsible AI with clear principles and policies
across focus areas like fairness, bias mitigation, ethics, risk management, privacy,
sustainability and regulatory compliance.
■ Consider responsible AI when selecting and prioritizing GenAI use cases. Analyze
each use case against your principles and policies, filtering out use cases that pose
an unacceptable risk level.
■ Evaluate and test emerging tools to mitigate GenAI risks, leveraging native tooling
from model providers, but also looking for solutions to augment their capabilities
(see Innovation Guide for Generative AI in Trust, Risk and Security Management).
This includes developing highly technical skills required to operate and customize these
tools, and awareness and capabilities across the business to effectively use and integrate
GenAI into business workflows. This transformation will require adopting change
management best practices to successfully drive adoption at scale such as:
■ Partner with HR to set up career mapping clinics and open mic sessions to address
the fear, uncertainty and doubt (FUD) that exists around AI’s impact on skills and
jobs. Consider the impact of different GenAI initiatives on employee workflows.
Involve the impacted employees upfront, identify their concerns and the barriers to
implement the change required, and maintain continuous engagement with them.
■ Focus on upskilling the technology teams with GenAI-specific skills in areas such as
prompt engineering, model validation and tuning, infrastructure management and
responsible AI — to name a few. Encourage citizen prompt engineering councils to
upskill and inculcate prompt engineering best practices.
GenAI models deliver the most value when combined with organizational data. This
presents some new and unique requirements when it comes to data management, such
as creating data pipelines for retrieval augmented generation, and utilizing tools like
vector databases and knowledge graphs to help organize and find relevant data. CTOs
should:
■ Instill good data engineering practices to ensure a high-quality and automated data
pipeline is available to customize and steer GenAI applications (see 5 Ways to
Enhance Your Data Engineering Practices).
■ Focus on the data that is likely to be used in your use cases, and improve it along the
dimensions required to enhance the performance of these initiatives. Do not try to
improve all of your data.
■ Train AI teams on best practices for integrating models with enterprise data via
vector embeddings as well as emerging approaches for efficient fine-tuning.
■ Ensure that GenAI applications — whether they are customer-facing or internal — are
designed in a human-centered way, integrated tightly into their workflows, are
intuitive to use and understand human intent clearly.
■ For your most critical use cases, ensure that there is a “human in the loop” to vet the
output of GenAI applications while exploring ways to automate the process to
ensure scaling. Collect, measure and use human feedback to improve the quality of
GenAI applications.
Ensure you clearly understand the pricing model of GenAI vendors well. Most application
vendors charge on a per user basis; they may subsidize or even offer these for free, but it
may not stay free or at a low cost forever. When you fine-tune or use commercial AI
models, you pay per token or batch of tokens used, which can lead to a significant
operational expenditure, if ungoverned.
To control costs:
■ There is an explosion of GenAI models today with each having varying capabilities,
token limits and costs. By pursuing a sandbox environment with multiple models,
builders can select the most suitable model for a specific use case.
■ Educate users on effective prompting techniques that can help reduce input and
output tokens. Consider prompt templates that can standardize effective and
efficient prompts. Explore prompt caching, so that queries can be answered without
the need for an API call.
Furthermore, the swift uptake of GenAI applications by both consumers and enterprises is
driving a rapid evolution in end-user demands: they now expect nothing less than optimal
user experiences from existing GenAI applications. This dynamic environment demands
an agile delivery model that can respond to shifting market innovations, underscoring the
relevance of the product-centric approach in developing, deploying and maintaining GenAI
applications.
■ Adopt a product-centric approach to GenAI with future plans for dedicated product
owners, continuous updates and regular assessments of current approaches’
effectiveness (potentially benchmarking them against innovative approaches
surfacing in the market).
■ Ensure product managers gather and measure user feedback to iterate the product,
enabling early detection and addressing of issues, thus mitigating any surprises.
■ Avoid running GenAI initiatives as static projects (rather than products), which
restricts the ability to adapt to fast-paced market innovations. This could lead to
outdated solutions and missed opportunities due to not addressing user feedback
and emerging trends.
1
Generative AI Realities: Proactive Approaches for Quantifiable Business Results — This
webinar was held on 14 September 2023 with 1,419 respondents to the polling. Results of
this poll should not be taken to represent all executives, as the survey responses come
from a population that had expressed interest in GenAI by attending a Gartner webinar on
the subject.
Establish a Continuous Process to Prioritize Use ■ Use the Gartner AI opportunity radar to set the ■ Involve the business units and functions that
Cases AI ambition and use-case prism template to will be the primary users of the use cases to
rank your chosen use cases. assess their value.
■ Create a framework to measure and track ■ Involve IT and data science teams to assess the
business value, both during pilot and in use case feasibility.
production.
Create a Decision Framework for Build Versus Buy ■ Develop a systematic decision framework to ■ Engage the sourcing team for input on
assess build versus buy options for each approved vendors and potential solutions.
potential GenAI use case within the
■ Involve IT and data science teams to assess
organization.
feasibility of development and deployment
■ Regularly revisit and update your categorization options.
of use cases across various approaches, given
the rapid evolution of techniques in the market.
Pilot Use Cases for Scalability ■ Adopt an agile mindset and run pilots to ■ Engage with data science, IT, security and
experiment with GenAI applications, using the business teams to form a multidisciplinary tiger
lessons to refine strategies. team to pilot and test GenAI use cases.
Design a Composable GenAI Platform Architecture ■ Develop a flexible and modular GenAI platform ■ Engage with IT infrastructure and data science
architecture that is scalable and incorporates teams to ensure the right balance between an
governance from the outset. end-to-end platform and a disaggregated stack
of best-of-breed components.
■ Invest in AI engineering tools that are model-
agnostic but tightly integrated, ensuring
efficient data integration, automated
deployment and responsible AI usage.
Put Responsible AI at the Forefront of Your GenAI ■ Incorporate a clear responsible AI framework ■ Collaborate with legal and compliance teams to
Efforts into your GenAI strategy, focusing on areas like identify potential risks associated with the use
fairness, bias mitigation, ethics and risk of GenAI, and create action plans for mitigation.
management.
■ Appoint an AI champion or involve a team
■ Regularly evaluate and test emerging tools to responsible for ensuring the ethical
mitigate GenAI risks, augmenting capabilities of development and usage of AI for each GenAI
model providers. use case.
Invest in Data and AI Literacy ■ Invest in initiatives to amplify AI literacy ■ Collaborate with the HR department to organize
throughout the organization for informed interactive sessions addressing AI’s impact on
utilization of GenAI. skills and jobs, and for career mapping.
Instill Robust Data Engineering Practices ■ Implement robust data engineering practices to ■ Partner with the data and analytics team to
ensure availability of high-quality, curated and focus on improving relevant data likely to be
well-governed organizational data. used in your use cases.
Enable Seamless Collaboration Among Humans ■ Ensure that GenAI applications are designed in ■ Partner with HR and business units to clarify
and Machines a human-centric way, capturing your intent and the rules governing human-machine
being integrated into your workflows. collaboration, and create strategies to
incentivize a meaningful partnership.
■ Collect, measure and use human feedback to
improve the quality of GenAI applications.
Apply FinOps Practices to GenAI ■ Adopt FinOps practices for GenAI to gain ■ Involve IT and data science teams to identify
visibility into costs, optimize strategies and techniques to actively manage and reduce the
manage expenses actively. costs.
■ Implement monitoring tools and education ■ Engage with existing FinOps cross-functional
methods to drive cost-efficient usage of GenAI teams to leverage their knowledge and
Adopt a Product Approach for GenAI ■ Implement a product-centric approach for ■ Engage with the AI center of excellence or R&D
GenAI with future plans for dedicated product team for monitoring GenAI market trends.
owners and frequent updates to improve the
■ Engage with the business units or functions
end-user experience.
where the application is in use, whether
■ Proactively monitor emerging trends to future- internally to employees or externally to clients,
proof your GenAI strategies and implement to gather feedback and suggestions for
innovative techniques that could deliver improvement.
immediate enhancements.