John lewis lego craig sullivan 1335
Download as PPTX, PDF1 like695 views
This document contains tweets and commentary from an analytics expert discussing A/B testing and optimization. It lists common mistakes in A/B testing like not testing for long enough, peeking at results prematurely, and failing to properly integrate analytics. It also provides advice on prioritizing opportunities, knowing when to stop tests, and the importance of testing whole purchase cycles. Throughout it emphasizes the importance of a collaborative, data-driven approach to optimization.
John lewis lego craig sullivan 1335
- 1. Oh Boy! These A/B tests appear to be bullshit!
- 2. @OptimiseOrDi e • UX and Analytics (1999) • User Centred Design (2001) • Agile, Startups, No budget (2003) • Funnel optimisation (2004) • Multivariate & A/B (2004) • Conversion Optimisation (2005) • Persuasive Copywriting (2006) • Joined Twitter (2007) • Lean UX (2008) • Holistic Optimisation (2009) Was : Working at Spareroom.co.uk Now : Doing workshops & consulting again!
- 4. Oh Boy! These A/B tests appear to be bullshit!
- 5. Top F***ups for 2014 1. Testing in the wrong place 2. Your hypothesis inputs are crap 3. No analytics integration 4. Your test will finish after you die 5. You don’t test for long enough 6. You peek before it’s ready 7. No QA for your split test 8. Opportunities are not prioritised 9. Testing cycles are too slow 10. You don’t know when tests are ready 11. Your test fails 12. The test is ‘about the same’ 13. Test flips behaviour 14. Test keeps moving around 15. You run an A/A test and waste time 16. Nobody ‘feels’ the test 17. You forgot you were responsive 18. You forgot you had no traffic 19. You ran the wrong test type 20. You didn’t try all the flavours of testing slidesha.re/1wBbZ9c @OptimiseOrDie
- 10. Oppan Gangnam Style! @OptimiseOrDie
- 11. @OptimiseOrDie
- 12. @OptimiseOrDie
- 13. @OptimiseOrDie
- 14. The 95% Problem • Many people use 95, 99% ‘confidence’ to stop • This value is unreliable • Read this Nature article : bit.ly/1dwk0if • You can hit 95% early in a test • If you stop, it could be a false positive • Tools need to be smarter about inference • This 95% thingy – it’s last on your list for reasons to stop testing @OptimiseOrDie
- 15. Business & Purchase Cycles @OptimiseOrDie Start Test Finish Avg Cycle • Customers change • Your traffic mix changes • Markets, competitors • Be aware of all the waves • Always test whole cycles • Minimum 2 cycles (wk/mo) • Don’t exclude slower buyers
- 16. 16
- 17. When to stop? • Know the cycles • Run a test length calculator - bit.ly/XqCxuu • Set your test run time • Yup, just like a retail store • Run it • Stop it • Analyse the data • That’s it. No peeking and waiting for the numbers to look good so you can stop. • Go again @OptimiseOrDie
- 18. No QA testing for the AB test?
- 19. QA Test or Die! • Over 40% of tests have had QA issues. • It’s very easy to break or bias the testing Browser testing www.crossbrowsertesting.com www.browserstack.com www.spoon.net www.saucelabs.com www.multibrowserviewer.com Mobile devices www.appthwack.com www.deviceanywhere.com www.opendevicelab.com Article bit.ly/1wBccsJ @OptimiseOrDie
- 20. Gamble the Company AWAY! • I get 60-70% right • UX and Copywriters good at picking! • C level execs are easy marks • Ironically, many decide ‘designs’ • You need collaborative test design • It’s a team game, with customers • Flip a coin, anyone?
- 21. WE’RE ALL WINGING IT
- 22. 2004 Headspace What I thought I knew in 2004 Reality
- 23. 2014 Headspace What I KNOW I know Me, on a good day
- 27. 23 : Business Future Testing? “Congratulations! Today you’re the lucky winner of our random awards programme. You get all these extra features for free, on us. Enjoy.”
- 28. #1 Smart Talented Polymath People The 5 Legged Optimisation Barstool @OptimiseOrDie Flexible and Agile teams
- 30. #2 : Analytics Investment (tools, people, dev time) @OptimiseOrDie
- 31. @OptimiseOrDie #3 : User research and insight
- 32. #3 : THE BEST IDEAS COME FROM? @OptimiseOrDie
- 33. #4 : GREAT COPYWRITING “On the average, five times as many people read the headline as read the body copy. When you have written your headline, you have spent eighty cents out of your dollar.” David Ogilvy “In 9 years and 40M split tests with visitors, the majority of my testing success came from playing with the words.” @OptimiseOrDie
- 34. • Google Content Experiments bit.ly/Ljg7Ds • Optimizely www.optimizely.com • Visual Website Optimizer www.visualwebsiteoptimizer.com • Multi Armed Bandit Explanation bit.ly/Xa80O8 • New Machine Learning Tools www.conductrics.com www.rekko.com @OptimiseOrDie #5 : Split Testing Tools
- 35. The 5 Legged Optimisation @OptimiseOrDie Barstool #1 Culture & Team #2 Toolkit & Analytics investment #3 UX, CX, Service Design, Insight #4 Persuasive Copywriting #5 Experimentation (testing) tools
- 36. READ STUFF
- 37. READ STUFF
- 38. READ STUFF
- 39. #5 : FIND STUFF @OptimiseOrDie @danbarker Analytics @fastbloke Analytics @timlb Analytics @jamesgurd Analytics @therustybear Analytics @carmenmardiros Analytics @davechaffey Analytics @priteshpatel9 Analytics @cutroni Analytics @avinash Analytics @Aschottmuller Analytics, CRO @cartmetrix Analytics, CRO @Kissmetrics CRO / UX @Unbounce CRO / UX @Morys CRO / Neuro @UXFeeds UX / Neuro @Psyblog Neuro @Gfiorelli1 SEO / Analytics @PeepLaja CRO @TheGrok CRO @UIE UX @LukeW UX / Forms @cjforms UX / Forms @axbom UX @iatv UX @Chudders Photo UX @JeffreyGroks Innovation @StephanieRieger Innovation @BrianSolis Innovation @DrEscotet Neuro @TheBrainLady Neuro @RogerDooley Neuro @Cugelman Neuro @Smashingmag Dev / UX @uxmag UX @Webtrends UX / CRO
- 40. #5 : LEARN STUFF @OptimiseOrDie Baymard.com Lukew.com Smashingmagazine.com ConversionXL.com Medium.com Whichtestwon.com Unbounce.com Measuringusability.com RogerDooley.com Kissmetrics.com Uxmatters.com Smartinsights.com Econsultancy.com Cutroni.com www.GetMentalNotes.com
- 41. #12 : The Best Companies… • Invest continually in analytics instrumentation, tools, people • Use an Agile, iterative, cross-silo, one team project culture • Prefer collaborative tools to having lots of meetings • Prioritise development based on numbers and insight • Practice real continuous product improvement, not SLEDD* • Are fixing bugs, cruft, bad stuff as well as optimising • Source photos and content that support persuasion and utility • Have cross channel, cross device design, testing and QA • Segment their data for valuable insights, every test or change • Continually reduce cycle (iteration) time in their process • Blend ‘long’ design, continuous improvement AND split tests • Make optimisation the engine of change, not the slave of ego * Single Large Expensive Doomed Developments
- 42. THE FUTURE OF TESTING
- 43. Thank You! 43 @OptimiseOrDie Email Slides : [email protected] : slideshare.com/sullivac : linkd.in/pvrg14