Menu Get In Touch

The state of experimentation in North America: 3 key insights and a path forward

Vantages

June 15, 2018

Experimentation is a way of doing business – it isn’t a 6-month project where you can say you’ve optimized your site.

Today, leading companies understand that experimentation is much bigger than A/B split testing on a landing page.

Organizations like Amazon, Netflix, Google, Walmart, and Microsoft are leveraging experimentation as a foundational growth strategy, testing across the customer journey. And other companies are rapidly following suit to avoid becoming obsolete.

GO > North America, together with its partners at Optimizely, surveyed marketers, product managers, and growth strategists at some of North America’s leading brands like Nike, United Airlines, Showtime, American Express, Hotwire.com, MailChimp and many more.

We summarize the maturity levels and share three key findings from the report by GO > North America. Lastly, we highlight an experimentation model global companies should aim for in their path toward digital transformation.

Most respondents are still only scratching the surface when it comes to experimentation.

Yet, many of these same organizations plan to accelerate the pace of experimentation in the next 12 months.

Interestingly, the percentage of companies not yet ‘scaling’ and ‘driving’ experimentation programs mirrored the level of digital maturity discovered by Deloitte Digital together with MIT Sloan School of Business.

Neanderthal or ‘sapien?’ Mother nature runs a relentless and ruthless experimentation program

Familiarize yourself with these stages of experimentation to better assess the urgency of establishing an experimentation program in your organization.

Don’t be surprised to find your organization lagging. Half of the respondents in the GO > North America survey are still in the first two stages of maturity, and 33 percent are in the third stage, ‘collaborating.’ None of the respondents in the report have reached the highest level of maturity: “driving.”  

Maturity levels of experimentation programs

Chart of GO North America Maturity Levels

Level 1: Initiating: Organizations at this stage are just getting started. An optimization champion is working to get initial wins to prove the value of an experimentation program.

Level 2: Building: In this stage, an organization is bought-in on the value of experimentation and an optimization champion or team is establishing a process and building the infrastructure to scale the program.

Level 3: Collaborating: Organizations at this stage are expanding the experimentation program and collaborating across teams. Finalizing a communications plan and overall protocol for the program is a priority here.

Level 4: Scaling: Experimentation is a core strategy for these organizations. Standards are in place and success metrics are aligned with overall business goals, enabling testing at scale.

Level 5: Driving: The highest level of maturity. Experimentation is the organization’s growth and product strategy. The Amazons, Netflixs, and Booking.coms are here.

The majority of mature organizations ‘driving’ experimentation have clearly defined and communicated metrics set, pointing to strong senior-level buy-in.

When looking at the total, however, less than 50 percent of those organizations surveyed have metrics set at the top level.

A successful experimentation program involves the entire executive team in the creation of success metrics for your experimentation program. Experimentation is linked to the goals executives know positively impact business. The program has visibility at the highest level of the organization.

Companies ‘driving’ experimentation programs are most likely to deploy and run hybrid operations that combine the best of a decentralized and centralized setup.

An experimentation program at an organization that is at the ‘initiating’ or ‘building’ stage is likely decentralized.

At these early-stage organizations, there is no central oversight of the experimentation program at a local or headquartered-level. Everything is run independently.  

While a decentralized program may enable speed of testing, without proper standards and central oversight, there is a risk that teams will clash, cannibalize each other’s goals, and pull in different directions rather than pushing the organization in a unified direction.

A central body will most likely have the holistic customer journey in mind, and can prioritize and deliver experimentation campaigns that focus on the most important parts of the business. But it’s unwise to assume a central body will have the necessary depth of expertise to experiment on certain parts of the business or take into consideration the diversity of regional and international inputs.

58 percent of the most mature organizations use a combination organizational model for experimentation.

In the combination model, there is shared ownership of digital initiatives – a central body works with all relevant stakeholders to prioritize and plan for experimentation. But individual teams are able to own and execute tests.

Highly mature and agile organizations rely on data to augment strategic decision making and make delivering a positive customer experience a top priority. These evolved companies are very likely to run a ‘combination model’ of experimentation.

In these hybrid setups, a centralized centre of excellence – steered by senior executives –  collaborates with experimentation leadership internally and externally e.g. international offices, etc. to establish, monitor, and troubleshoot the company’s experimentation programs. Individual teams are able to own and execute tests. The agile setup helps contributes to the company’s overall digital growth and performance.

Companies that have successfully implemented this combination model are distinguished leaders in their industry e.g. Airbnb.

Organizational structures for experimentation programs

Experimentation programs often fall into one of the following organizational structures below.

Decentralized (All-owned testing): Optimizers exist in different teams across the organization. Each strategizes and executes experiments according to their own KPIs, as they are positioned to reach larger organizational goals. This model does not have the oversight of a central team.

Centralized (Center of Excellence): Often in companies where individual teams have their own sites or domains. A central body that owns experimentation across the business, encouraging the growth of skills and the expansion of the program.

A Combination Model: When there is shared ownership of digital initiatives, a central body works with all relevant stakeholders to prioritize and plan for experimentation. But individual teams have the ownership to execute tests.

There are pros and cons to each experimentation program structure, executive leadership must figure out what will work best for their organization and their unique culture.

Towering tree trunk

Four measuring sticks to evaluate the maturity of an experimentation program

Process & Accountability: The most mature organizations keep process and accountability at the core of their experimentation strategy, fuelling how experiments are developed, and results are analyzed, understood, and leveraged.

This pillar includes an organization’s experimentation protocol and methodology, process for ideation and prioritization, experiment design, and measurement of success.

Culture: Culture is crucial when defining experimentation maturity: Does your organization celebrate testing and learning? Are people encouraged to try (and fail) and try again?

This pillar includes organizational buy-in for experimentation, program support from the C-level, and cross-team participation in an experimentation program.

Expertise: An experimentation program needs expertise and resources. The amount of time and full-time team members dedicated to experimentation is reflective of an organization’s maturity.

This pillar includes people and skill sets: strategists, analysts, designers, developers, project managers, product owners, third-party partners, as well as hours dedicated to experimentation.

Tools & Technology: Experimentation maturity requires a well-rounded technology stack. Experimentation and personalization tools, visitor engagement tools, customer data tools, project management tools.

Mature organizations have the right tools in place to ensure they can develop the best possible hypotheses and have reliable data.

 

Steady advancements in technology and computing power are destabilizing bedrock companies and brands, today. As a result, the importance and urgency for incumbent leaders to accelerate their digital transformation by harnessing the power of experimentation remain high. Although many of these companies in North America are still in the infancy stage of building an impactful experimentation program, fortunately, there are clear organizational models and evaluation pillars to provide a path forward.

GO Logo_GoIcon

This article was adapted from GO > North America’s, ” 6 key insights from the “State of Experimentation Maturity 2018” research report” by Lindsay Kwan. Read the full report by GO > North America to accelerate your digital transformation evolution.

Close

Ready to go evolutionary to epic?