Optimizely Product Vision: The Future of Experimentation

Editor's Notes

• #3: My name is Adam…. I’m excited to be joined by Olivia... Olivia leads the digital testing for Wall Street Journal.com and Barrons.com. She’s helped inject testing into all aspects of the customer lifecycle, including acquisition, engagement, product, and retention. Her team has run over 500 experiments over the past 3 years and is continuing to find new ways drive the product roadmap and expand acquisition strategies.
• #5: A few housekeeping items for today: We are recording this webinar You’ll receive the recording and slides in a follow-up email and we want today’s session to be interactive. So please ask questions. You can do so by entering them in the chat box of Go to webinar console.
• #6: Many of the world’s greatest companies have already broken the code on how to win in the new world order of digital. The answer for them is experimentation. Companies like Amazon, Google, Facebook and others are all saying exactly the same thing. They have figured out the experimentation is the key to winning in an increasingly digital world. Jeff Bezos at Amazon is widely quoted about how their success is directly tied to the number of experiments they run per day. And as you can see here, very similar comments from Larry Page, and Mark Zuckerberg And if that’s not enough, even the Harvard Business Review is weighing in. In a recent article called the ”Surprising Power of Online Experiments” the authors stated that “if you want to achieve a competitive advantage, you need to master the practice of online experiments.” So, yes, we do believe experimentation is the next big thing in business management and is rapidly becoming a mainstream business best practice.
• #7: Let’s talk about why this is important now more than ever. These stats are to drive home that consumer behavior has fundamentally changed. Ten years ago we bought physical CDs and DVDs, shopped in brick-mortar-stores, and if we needed to file an insurance claim, we called into a call center. Now all of that is digital. Digital has shifted from being a convenient way for consumers to engage brands to being the central way.
• #8: And this shift is not something just affecting marketing. Every single touchpoint - across web, mobile, voice - along the entire customer journey is an opportunity to win or lose your customer. We see everyday more and more companies adopting digital experience optimization across all business functions because it is key to acquiring more users, driving more and deeper product engagement, and retaining more customers.
• #10: Complete coverage of digital channels so you can optimize everywhere your users are The ability to optimize and personalize the entire experience from the front-end UI to server-side business logic and data Trusted results so you can make sound decisions quickly Consumer-grade usability to scale experimentation enterprise-wide Protection of your user data through enterprise grade security & compliance Efficient ways to govern and manage your optimization programs at scale - the more you test and learn, the faster you grow A strategic partner with the hard-won best practices, knowledge and services to build with you into a world-class optimization program.
• #11: Complete coverage of digital channels so you can optimize everywhere your users are The ability to optimize and personalize the entire experience from the front-end UI to server-side business logic and data Trusted results so you can make sound decisions quickly Consumer-grade usability to scale experimentation enterprise-wide Protection of your user data through enterprise grade security & compliance Efficient ways to govern and manage your optimization programs at scale - the more you test and learn, the faster you grow A strategic partner with the hard-won best practices, knowledge and services to build with you into a world-class optimization program.
• #12: Complete coverage of digital channels so you can optimize everywhere your users are The ability to optimize and personalize the entire experience from the front-end UI to server-side business logic and data Trusted results so you can make sound decisions quickly Consumer-grade usability to scale experimentation enterprise-wide Protection of your user data through enterprise grade security & compliance Efficient ways to govern and manage your optimization programs at scale - the more you test and learn, the faster you grow A strategic partner with the hard-won best practices, knowledge and services to build with you into a world-class optimization program.
• #13: Complete coverage of digital channels so you can optimize everywhere your users are The ability to optimize and personalize the entire experience from the front-end UI to server-side business logic and data Trusted results so you can make sound decisions quickly Consumer-grade usability to scale experimentation enterprise-wide Protection of your user data through enterprise grade security & compliance Efficient ways to govern and manage your optimization programs at scale - the more you test and learn, the faster you grow A strategic partner with the hard-won best practices, knowledge and services to build with you into a world-class optimization program.
• #14: Complete coverage of digital channels so you can optimize everywhere your users are The ability to optimize and personalize the entire experience from the front-end UI to server-side business logic and data Trusted results so you can make sound decisions quickly Consumer-grade usability to scale experimentation enterprise-wide Protection of your user data through enterprise grade security & compliance Efficient ways to govern and manage your optimization programs at scale - the more you test and learn, the faster you grow A strategic partner with the hard-won best practices, knowledge and services to build with you into a world-class optimization program.
• #26: Optimizely has been the site experience laboratory for the better part of a decade. After examining thousands of tests and working with thousands of customers, providing a relevant experience based on previous behaviors and profile data tends to lead to a higher desired conversion rate. We have learned that the winning experience is not the same for everyone. The right experience really is dependent on context and that the right experience requires continuous iteration. Those that have really excelled at this often have not gotten that conversion rate on their first try and the way that they have gotten this is by experimenting every step of the way, different audience definitions, different experiences/variations and ultimately treating every decision as a test
• #27: Let’s look a bit at what we mean when we say that the right experience is not always the same for everyone. We see that different experiences resonate with different individuals, personalization enables you to ensure you are delivering experiences that resonate with your customers. Let’s walk through an example +++Click Here we start out with an A/B test that everyone sees. We can see that Banner B does not do well in this test. +++Click However, when we dig into this and look at segments, we can see that while Banner B didn’t resonate with everyone, it does resonate better than the alternative with students. +++Click From this information we then decide to experiment and apply an experimentation mindset to this banner, optimising for the student segment, and actually find that banner d is the winner (+++Click) This is an example of how we can see that by applying an experimentation mindset, we are able to effectively prsonalize.
• #30: Trend : product experimentation
• #57: As we talk with companies like yours we find that they are seeking ways to accelerate product innovation to meet rising customer expectations, and reach the market faster in order to see immediate impact to revenue.   However, we also know that adding velocity greatly increases the associated risk. That’s why leading companies adopt processes like agile and devops—to mitigate risk while moving quickly. And finally, these companies also experiment on everything, so they can know the impact of new features on key metrics before rolling out to everyone.
• #58: So what does the infrastructure that backs this process look like when implemented at scale? It starts with feature flags, simple toggles that enable you to separate deploying code from enabling features for customers. Then these teams do staged rollout of new features, to a small percent of traffic to ensure things work as expected, and can safely roll back if they don’t. Features are also A/B tested to quantify impact on key business metrics, not just performance. As these companies iterate, they perform many more A/B/n or multivariate tests to try out different variations on a feature or experience and tune to perfection. Every feature goes through this process before launch. And it’s deceptively simple. There’s a lot under the hood that these companies have built, including: Robust data infrastructure – statistical analysis, targeting Tools to democratize testing across the product org – a usable interface for business users, support for many languages, documentation for onboarding more teams Enterprise ready capabilities—is the system secure, compliant, and available when they need it? And integrations to other systems that power the customer experience. When you look at the multitude of discrete components that make up a robust experimentation stack, it's a daunting task to undertake.
• #61: As we talk with companies like yours we find that they are seeking ways to accelerate product innovation to meet rising customer expectations, and reach the market faster in order to see immediate impact to revenue.   However, we also know that adding velocity greatly increases the associated risk. That’s why leading companies adopt processes like agile and devops—to mitigate risk while moving quickly. And finally, these companies also experiment on everything, so they can know the impact of new features on key metrics before rolling out to everyone.