Incorporating AI into 30,000 employees’ most critical and beloved tool
3 min read
Accountants use Excel, designers use Figma, and consultants use PowerPoint.
Improving output and increasing productivity for a professional services firm is a huge opportunity. While many off-the-shelf tools will help productivity, the creation and communication of strategy work and change management will remain at the core of management consulting. And that means slides.
CONTEXT
As firms adopt AI-enabled tools across the enterprise, one area where an off-the-shelf tool won’t cut it is in the day-to-day content authoring that consultants do in PowerPoint. Our team had the opportunity to create a PowerPoint assistant for a global consulting team. While many productivity tools are already being commoditized, the creation of content based on analysis, client context, and the firm’s existing knowledge base is how consultants communicate value to their clients.
If we can tap into the firm’s collective knowledge and IP to deliver insightful thought partnership and speedy production assistance, can we improve productivity while also improving the quality of content created by 30,000 consultants?
WHAT WILL BE MEASURED?
Adoption of new ways of working in PowerPoint
Quality of users’ output
Perceived time savings
OUTCOME
Currently, the beta version of the assistant is in a pilot program and initially launched to 300 users in October 2023. Since then, as CSAT scores have trended up, word-of-mouth access requests have created a long waiting list and we’ve scaled access to nearly 3,000 users, while maintaining good usage metrics and CSAT scores. Global roll out to over 30,000 employees is scheduled for Q2 2024.
HOW’D WE DO IT
Easy proof of concept, hard real product
The team started very small with a 3 week sprint to demo. The demo was at a global gathering of the firm’s partners. As tough of a crowd as that was, it went over well, including catching the CEO’s eye for a personal demo and discussion.
Getting from that proof-of-concept to a valuable tool, though, was a challenging, winding road to navigate. To integrate GenAI into a product that is used for hours each day—if not the entire day—is a high bar to clear for user adoption. Custom workflows, shortcuts, and button layouts are the norm for PowerPoint power users in consulting. Breaking through that meant that value and usability needed to be sky high—and work well for both first-time users and power users as they integrate it into their workflow.
Tech showed what was possible, design & research set the direction
As we got traction early on, the excitement from leadership was growing. This also meant that imaginations were in overdrive. A mix of incremental feature improvements and fully automated agent-based solutions were dreamed up every time a senior stakeholder was in the room with our team. And the quickly evolving tech capabilities made all of these seem just barely out of reach—and closer by the day.
I led the team in developing a framework to organize our user journeys around Finding, Creating, and Refining. This was valuable in both managing up to stakeholders as we communicated our priorities, and to focus our ongoing user research efforts around these pillars of consultant workflow.
New ways of working to adapt to evolving tech capabilities
Most of the technology we’re using is novel to the entire team—even those who had been using LLMs pre-ChatGPT are still experiencing the newest capabilities in real time with everyone else.
Because of the emergent capabilities of these models combined with the relative immaturity of our product, we quickly started to adopt several new ways of working. These varied from our typical product design and development, where things are often more predictable.
KEY LEARNINGS FOR GENAI PRODUCT DEVELOPMENT
-
Things move extremely fast, which broke our “design-test-build” pattern. As ideas came from users, leaders, engineers, and everyone else, even our small team had to separate experiments from develop tasks. Many items were explored for a day or a week before being tossed or incorporated into roadmap. Feasbility has never been harder to pin down than when working with generative AI in an authoring environment like PowerPoint.
-
In parallel to the blurred line between discovery, exploration, and delivery, we changed how we would typically run a design critique to take advantage of the broader team’s explorations. We renamed it a Product Critique, as we opened the agenda for the everyone to bring their work in progress for discussion. This meant everything: developing slide templates and best practices, the latest prompt engineering tests, and sandboxed engineering work that explored new ways of rendering content in PowerPoint. These are some of the best collaborative sessions of my career—where everyone truly is a designer.
-
For this type of multi-modal interaction design and content generation, evaluating the quality of output is quite a unique challenge. One bottleneck is that only developers can define the prompts, test the output, and tweak the instructions in a tight loop. To break this down, I spearheaded the effort to create a non-technical tool for anyone to tweak and test new prompts for our features. That way, the strategy consultants on our team could make a ton of progress in improving our prompt design without taking away our engineering team’s development time.
-
A new way of thinking about testing is crucial to developing and scaling GenAI tools. In this case, balancing trust, capability, and usability was our focus. These three pillars meant that we needed more user involvement throughout the process—and more involvement of non-designers to capture the opportunities and evaluate what was successful. An engineer is not as good at judging slide output as a consultant, and a designer might not understand the possibilities as well as the data scientists when we run into a shortcoming.
These are a few of the core principles I’ve seen so far emerging as enterprises and product teams adapt ways of working to new technological possibilities. It hasn’t felt this new an exciting to be a person in product design and development in a long time.