Digital experimentation experts reveal what separates a great experimentation program from one teetering toward collapse
Too often, successful digital experiments do not get implemented into a business. This is odd. Why wouldn’t a change that is proven to reward customers be put into motion, especially if the evolution improves the company’s bottom line?
When a positive customer-centric change isn’t enabled in a company, the explanation is simple: the experiment had little to no connection to a clarified business purpose.
Examples of program failure due to poor business alignment
- A major telecom brings in a big four consultancy and builds a center of excellence but after six months cannot launch a single experiment.
- An international SaaS company tracks conversions but can’t calculate a customer’s lifetime value by segment
- The executive board of an insurance group buys not one, but two testing platforms – for a single market – but can’t determine if the program is producing any meaningful impact.
Buy-in doesn’t mean issuing an edict from the corner suite.
In all three of these real business examples, management was traditional, what drove the business forward was poorly understood, customer solutions were siloed, and there was no culture or infrastructure to reward or support knowledge exchange. Innovation via experimentation was set to fail.
Businesses with experimentation programs grow faster.
The constant implementation of customer-validated business changes across digital properties is a powerful method for organizations to innovate and transform digitally. Businesses listen to their customers via qualitative and quantitative data and test ideas to better satisfy and reward their customers. The concept is simple and compelling, which is why 89 percent of businesses say that experimentation is important to transforming the digital customer experience.1. Leaders at these companies appreciate that organizations that can create a healthy culture of experimentation grow at least eight times faster than global GDP.2.
Organizations that can create a healthy culture of experimentation grow at least eight times faster than global GDP.
Agile work environments are crucial for experimentation programs to blossom.
Building a culture of experimentation that reflects across the C-suite to tactical units is so important that companies that can’t build agile environments do not grow as fast as those who do, and they are at a higher risk of future-failure.3.
Fortunately, there are distinct signals that self-aware companies can note to help them determine if their experimentation program is teetering towards failure. We asked world leaders in digital experimentation for these signs and compiled the top indicators.
The good news? Even if you see evidence of possible failure for your program, it doesn’t mean the program is doomed. There is a three-step plan you can take today to get your business harnessing the power of experimentation.
The experimentation program is not tied to an overarching business purpose and lacks a central, guiding metric
One of the quickest ways to run an experimentation program into the ground is to fail to link it to an overarching business purpose and unifying goal. “If by the first round of discovery, I’m not hearing a chorus of voices referring to the [experimentation] program’s North Star metric, I get spooked,” explains Chris Goward, CEO of WiderFunnel. “Without goal alignment from the very beginning, experimentation will go sideways quickly. You’ll see teams cannibalizing each other, prioritizing their own individual metrics over the big picture.”
“When a middle manager can confidently explain how their proposed solution connects to the company’s main revenue metric, like MRR [monthly recurring revenue], then you know you have a solid chance of having an impact,” adds Goward.
Teams are siloed and experiments do not consider the entire customer journey
A unifying North Star metric is one piece of the puzzle. Visibility into the entire customer journey is another.
“Successful experimentation programs are not led by conversion rate optimization (CRO) experts, they are led by ‘customer-journey’ or ‘growth’ leaders,” says André Morys, CEO of konversionsKRAFT in Germany. “Customers don’t care about optimizing form-load speeds. Those have to be fast. They care about finding their tax returns or getting approved for a loan quickly.”
“Silos are where innovation goes to die,” says Morys. “Insights come from ‘two-pizza teams’ tasked to take on a journey from A to Z.”4.
At the GO Group, experimentation teams are set up around end-to-end accountability
- Data analysts translate data accumulated from multiple customer touch points
- Behavioral scientists decode customer motivations and sentiments at every stage of the funnel
- CRO consultants process the data and customer research insights to propose novel customer-journey solutions
- UX designers breakdown the customer-journey solution into steps, frames, and processes
- Developers digitally engineer the solution and put it into QA; and
- A customer-journey team leader manages execution, ensuring that there are weekly check-ins with key stakeholders in the company and their customers.
Experiments have to be perfect before they see the light of day
The age of the waterfall project is over. Digital disruptors like Uber, Airbnb, Booking, and Amazon stay ahead of competitors in tiny sprints. Software is updated overnight, not every six months. This kind of speed and agility is achieved through trial and error. “You want to know what makes perfect? Practice,” says Nima Yassini, CEO of New Republique in Australia and New Zealand.
“We always say, ‘add a zero,’” says Morys. “That’s how many experiments you should be running.”
Companies that see a culture of vulnerability as a weakness will struggle, as it’s vulnerability that underpins risk-taking.
The trap of perfection is that while you’re working on perfection, your competitor is chatting up your customer,” says Yassini.
“Losing” experiments are disregarded rather than investigated
Executives are trained to look at red and think, bad. But this binary perspective can ruin an experimentation program. Losing experiments, as long as the owners can explain how the results revealed a customer insight, can have a much larger business impact than ones that win.
“If only a quarter of experiments ‘win’, you have to figure out how to get value from the other 75 percent,” explains Stephen Pavlovich, CEO of Conversion in the United Kingdom and Ireland.5.
“We all want winners but the bigger goal is to craft a nuanced understanding of the customer at every need state,” adds Pavlovich.
“Every experiment you run should be designed so that, no matter the outcome, you learn something,” says Goward. “Great programs add speed to the mix.”
“To create a lasting experimentation program, you need to come from a desire to learn first. Winning is an outcome of delivering an exceptional customer experience,” says Yassini6.
A lack of trust and a fear of failure cripple innovation
A healthy culture of experimentation is human-centric. It recognizes that individuals thrive when they:
- Believe their work has purpose
- Are trusted to deliver on that purpose; and
- Are held accountable (good and bad) for their actions.
“Most of your great ideas (hypotheses) are going to turn out to be wrong. So, if you’re worried about being seen as a failure, then it doesn’t matter how smart or senior you are, you just won’t share. And how does innovation thrive there?” says Natasha Wahid, head of content marketing for WiderFunnel.7.
“I can put all the right tools and tech in front of a very smart manager, but unless that individual feels their work will have an impact, then it’s often testing for testing’s sake,” says Morys.
Consider Booking Holdings, who own Kayak, Agoda, and several digital travel brands.8 The experimentation program at Booking, which took years to build is credited with helping propel the company’s considerable revenue year after year. Experimentation at Booking isn’t limited to a team, but to the democratization of action, where employees are given the ownership and creative freedom to validate their ideas. “We can’t give people the tools to run experiments, but not the ownership and accountability to make the [business] decision,” says Lukas Vermeer, head of experimentation at Booking.
There are ‘good’ ideas and then there are the ideas that you should test.
“Companies that see a culture of vulnerability as a weakness will struggle, as it’s vulnerability that underpins risk-taking,” says Wahid.
Building a culture that supports risk-taking and innovation isn’t limited to the experimentation program either; it radiates throughout the entire organization. Consider the following quote in the context of experimentation. It was extracted from a recruitment video at Netflix, “Freedom and responsibility to me is the gateway to creativity. When you have someone telling you what you can do and not what you can’t do. To me, that’s a very liberating framework in which to function.”9.
“You’ve gotta accept that your business is going to be in many ways an experiment, and it might fail,” says Jeff Bezos, founder and CEO of Amazon. “…You are going to fail a lot, and it’s OK, and you need to have a culture that supports that,” adds Bezos10.
Experiment ideation isn’t derived from listening to customers
“There are ‘good’ ideas and then there are the ideas that you should test,” says Goward.
Stellar experimentation programs make data-driven, customer-centric decisions by tapping into two data sources:
- quantitative sources where vast amounts of customer information can be analyzed to surface insights and ideas; and
- behavioral science, which maps out customer need states and motivations at every stage of the business funnel and proposes solutions to organize information accordingly.
“It’s not only listening, but also translating,” says John Ekman, the founder of Conversionista in Scandinavia. “And once you’ve got those messages translated, you begin to realize just how important it is to be curious and ask questions.”
“It’s not about whether the hero image should be a person smiling or if the button should be a different color,” says Morys. “It’s about whether the proposed change is based off a relevant customer insight.”
Every experiment you run should be designed so that, no matter the outcome, you learn something.
“It’s not a one-size fits model, either,” says Michael St. Laurent, head of optimization at WiderFunnel. “High traffic, mature websites in single markets can heavily rely on quantitative data to inspire new ways to reward and satisfy a customer. They can also validate that the proposed solution had a business impact faster than say a new consumer base, which lacks the ability to produce rich ‘quant’ data.”
“What we’re looking to help create is a program that understands the value in both data sets, as the data inspires the customer solution,” adds St. Laurent.
The program has no “library” of customer insights or feedback loop
In an engagement with a furniture retailer with offices in all major markets, a discovery revealed that separate experimentation teams from five different markets had tested the same customer solution. Worse, the solution resulted in no notable business uplift nor could the teams articulate what the results revealed about their customers.
“It’s hard enough managing multiple offices around the world much less coordinating knowledge exchange,” says Pavlovich. “But if there is no mechanism to record, recall, and share experimentation work, you’re doomed to repeat the same mistakes over and over,” adds Pavlovich. “Worse, the solutions that ‘won’ aren’t being considered for other markets.”
The easiest solution to scaling experimentation locally or internationally is to first agree on the data standards and structures you’ll record for every experiment. “You have to set the width of the railroad track before you can start sending the trains,” says Mirko Melcher, head of optimization at konversionsKRAFT.
“Defining standards also helps coordinate and troubleshoot the process of experimentation, not just the record keeping,” says Yassini. “No process; no program.”
Aim to be powered by data and driven by great people.
Programs need leaders and moments to celebrate customer insights.
Creating a library of experimentation results is only as worthwhile as the program’s determination to set up a feedback loop. Reviewing notable experimentation work not only helps evangelize the impact of experimentation; it also helps keep programs aligned to a business purpose – engaging stakeholders at regular intervals.
- Programs that scale and deliver valuable business results are supported by executives that organize and attend monthly knowledge exchange sessions.
- Department leaders who shape and affect the customer journey participate.
- Simple case studies break down and highlight the customer insights discovered inside the program and company. The feedback and peer review sharpen and drive the program toward excellence.
“The database is the easy part,” says Thorsten Barth, head of technology and co-founder of konversionsKRAFT. “The big value is showcasing the impact the work has had on the company’s bottom line.”
“It’s the programs where leadership is fighting to share their customer insights that I know have made the ‘transformation to digital,’” says Lulu Phongmany, head of business development for WiderFunnel.
In his book, “Faster: Demystifying the Science of Triathlon Speed,” author Jim Gourley shatters amateur athlete hearts. It turns out that a bike that is two to three pounds lighter (considerable weight when it comes to modern road and tri bikes) isn’t faster. What makes one go faster is increasing muscle strength, losing body weight, and practice, reveals Gourley.
It seems self-evident, but too often do organizations mistake technology for strategy. The consequences are enormous. Over USD 90 billion is reportedly wasted in failed digital transformation waterfall projects 11. “It’s a seductive message, buy me and I’ll solve your problems, but it’s only when the business strategy and culture of experimentation are ingrained in the organization that the technology begins to have an impact on performance, says Morys. “You would never pick up a hammer expecting it be a home,” adds, Barth.
“We hate to hear it, but it’s true; it’s the talent, not the tools,” says Pavlovich. “Aim to be powered by data and driven by great people.”
“The first tool I recommend is one that captures real customer behavior. Your customers’ needs should be driving your strategy and without knowing them, you’re guessing,” says Chris Formosa, head of partnerships at WiderFunnel.
“The solution isn’t for the board to rush out and hire a digital superstar to scale up a tool. Instead, leadership should be supported to articulate the value of the technology in helping the organization reach its business goal,” says Gursimran “Simar” Gujral, head of Optiphoenix in India.
A three-step action plan to harness the power of experimentation
Few experimentation programs launch and grow without complications. Regardless of maturity, there are course-correctors, or “steps” a business can take to ensure its program is set up to foster customer-centric innovation and growth.
1. Define, document, and disseminate your central, guiding North Star metric for business growth.
Defining an overall North Star metric, or better, an overall evaluation criterion (OEC) for your business is not easy—it will require serious thought and debate from key stakeholders. But it is necessary if you want to build a powerful experimentation program.
“When designing controlled experiments, one of the most important questions to ask early is: what are you optimizing for? Getting agreement on what the goal is and how to measure it is a huge step forward for the organization,” says Ron Kohavi, Technical Fellow and Corporate Vice President, Analysis & Experimentation at Microsoft.12.
If you don’t already have an overall evaluation criterion or North Star metric, you’ll want to consider the following when choosing one:
- Is it indicative of long-term gain in key performance indicators (KPIs)?
- Is it difficult to game? For example, if the metric only focuses on one part of the funnel, you may be able to drive ‘positive’ change by cannibalizing other parts of the funnel. This is what we want to avoid.
- Is it sensitive?
Examples of valuable North Star metric
- Disney judges its growth via customer lifetime value (CLV)
- Salesforce works to boost its monthly recurring revenue (MRR)
- Facebook relies on daily active users to judge company health
The following are NOT North Star metrics
- No. of inbound leads
- Basket size
- Check out rate
- Number of forms submitted
- Completion rate
- Conversion rate
2. Start with a minimum viable product (MVP) version of one customer journey, not the entire business.
Examples of a customer journey
- Providing documents to support tax filing
- Setting up an investment account
- Finding a local retailer
The “minimum viable product” version of one customer journey is a starting point; it will evolve. To build it, try to gather key stakeholders for two to three uninterrupted hours – “a two-pizza team.” The initial meeting should include product, marketing, sales, customer service and leadership representatives—the people in your organization who interact with your customers.
Everyone will likely have an idea of what this journey looks like, but you’ll want to synthesize these ideas into one map. Choose a strong enabler to lead the discussion. Based on this map, stakeholders should then design a cross-functional team that includes designers and developers to build the MVP.13.
Even a “small” customer journey has funnels and stages. Test and refine each stage of the MVP against the current journey.
“Your goal isn’t to be pixel perfect. It’s to get started learning about your customers,” says Morys. “Design workflows and teams around their journeys.”
3. Celebrate what you learned (not only the wins (or losses)) about your customer
Innovation requires taking risks, which means to expect failure. “Focus on the learning, not just the win,” says Yassini.
To signal that the company truly supports innovation, shine a light on ‘failures.’ This doesn’t necessarily have to be a losing experiment from the customer-journey MVP—it could be any bold idea gone wrong or a result that led to a surprising insight.
Present this ‘failure’ at the next quarterly meeting, or create a virtual space to share this failure. Communicate that your organization’s leaders will be sharing failures-turned-learnings consistently from this moment forward. Track whether this initiative leads to an increase in ideation across the organization.
“There’s an easy hack to stimulating creativity,” adds Yassini. “Tell stories about negative and surprising results. Set the example from the top-down. Use these moments to emphasize humility. Take it even further and reward well-thought out failures.”
Do not use the sharing opportunity to explain the process, tactics, or technicalities of the experiment.
Find the experimentation activators that will drive the program forward
“The goal of experimentation is not to be right, it’s to make a business impact, to power more confident decision making that results in action. To move from concept to action, you have to have experimentation ‘activators’ on your team,” says Ekman.
“Activators”, as Ekman calls them, make things happen. They are the individuals within an organization who can turn thoughts, ideas and concepts into action. They can translate experiment ideas, evidence and results to stakeholders at the executive level and across teams, generating buy-in and excitement as well as enabling individuals to act.
Your goal isn’t to be pixel perfect. It’s to get started learning about your customers
“These unicorns can be tough to find, but are critical to scaling experimentation. The best have a high EQ, business acumen, technical fluency, project management skills and are natural relationship builders,” says Morys.
Evolutionary to epic
Executives that can communicate a clear business purpose and enable the brightest talent to discover and activate the customer-insights that serve that purpose – at speed and scale – are set to succeed in tomorrow’s economy. “The goal really isn’t to build a successful experimentation program; it’s to build a successful experimentation company,” says Goward.
- Optimizely (May 2019) How to win in the digital experience economy
- Forrester Research (July 2016) Insights-Driven Businesses Will Steal $1.2 Trillion In 2020 — It’s Time To Join Them
- McKinsey (2018) Global Survey: Unlocking success in digital transformations
- The Guardian (April 24, 2018) The two-pizza rule and the secret of Amazon’s success
- GO Group Digital (May 2019) Centralized Repository of Experimentation Data (CRED): Win/Loss Ratio 32.34%
- Nima Yassini, New Republique (May 2019) Why most CRO programs are doomed to fail
- Natasha Wahid, WiderFunnel (April 16, 2019) Change organizational culture
- Booking Holdings (2019) Financial information
- Netflix (May 2019) Anna Lee, Director of Content Services at Netflix
- re:MARS (June 2019) Jeff Bezos: keynote session
- Harvard Business Review (March 2019) Digital Transformation Is Not About Technology
- Ronny Kahavi, Microsoft (2019) Chapters six – Advanced Topics in Experimentation
- Conversion (July 11, 2018) Introducing: The 9 experimentation principles