Remote, unmoderated usability and user testing.
3 likes466 views
This deck gives you an intro to unmoderated, remote user testing methods, latest tooling and ways to start.
Ad
More Related Content
What's hot (20)
Similar to Remote, unmoderated usability and user testing. (20)
Ad
More from Marc-Oliver Gern (18)
Ad
Recently uploaded (19)
Remote, unmoderated usability and user testing.
- 1. Unmoderated, remote User Testing. Get to know it better. 20/01/2018 © Marc-Oliver | Experience Lead
- 2. 1. What is unmoderated user testing? 2. Why we should do it? 3. How others do it 4. How we get started 2
- 3. What is unmoderated UT? _
- 4. What is unmoderated user testing Unmoderated usability sessions are completed alone by the participant with no real-time interaction or influence from another person. The main focus of such a study is a few specific elements, rather than an overall review. It’s great to see the impact of a minor change. The more ‘moving parts’ you need to test, the more you have to invest in technology and setup. This is also true for prototype versus testing on production.
- 5. Pros » Low running cost, once setup. » Can do last minute » Potentially with many participants » Easier to recruit given you have proper channels set up (Facebook, twitter, email etc) » Can go global on a large scale » Users use their own device and ‘natural environment’ Cons » Might need a lot of software/setup » No influence when something goes wrong » Cannot control precisely who does it and when » No follow up questions – harder to get the ‘WHY’ » Have to spend more time filtering the results » Sensitive data/design can leak easily
- 6. Why we should do it? _
- 7. Why you should do it testtest What you (probably) currently do time informed or semi-informed tinkering, tinkering, tinkering, tinkering, tinkering ….
- 8. Why you should do it What you should do: Frequently collect new small/relevant data points. Know what people do right now. Respond immediately. Test & Iterate Test & Iterate Test & Iterate Test & Iterate Test & Iterate Test & Iterate Test & Iterate Test & Iterate Test & Iterate time
- 9. How others do it. _
- 10. How Netflix does it Netflix often runs 20 to 30 ‘experiments’ at the same time on an ongoing basis, with different audiences (subscriber types) and in selected markets. They do this with a complex setup of tools and live (web)services. Resource: • https://medium.com/netflix-techblog/a-b-testing-and-beyond-improving-the-netflix-streaming-experience-with-experimentation- and-data-5b0ae9295bdf • https://medium.com/netflix-techblog/how-we-determine-product-success-980f81f0047e • https://medium.com/netflix-techblog/netflix-recommendations-beyond-the-5-stars-part-2-d9b96aa399f5 • https://medium.com/netflix-techblog/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429 • https://www.quora.com/profile/Neil-Hunt/answers/A-B-Testing • https://medium.com/netflix-techblog/optimizing-the-netflix-streaming-experience-with-data-science-725f04c3e834
- 11. How etsy does it etsy – an online retailer – is known for their “test first – don’t discuss” culture. They focus on multi-variant and A/B tests on their live website and can deploy tests in hours. Resource: • https://www.slideshare.net/InfoQ/ab-testing-continuous-delivery-everyday-product-launches?qid=66f85988-52fb-4d28-bece- b1b8c5465ab4&v=&b=&from_search=4
- 12. How Meetup does it There is never a question as to whether a product or (new/existing) feature should be tested at MeetUp … the only question is “how soon can we fit this in?” Resource: • https://www.youtube.com/watch?v=8QO-2olF9M4 • http://making.meetup.com/post/46268864442/how-meetup-hq-does-usability-testing • https://uxdesign.cc/meetup-a-usability-case-study-e909c33f1e3e
- 13. How Obama did it in 2008 Optimisation was the name of the game for the Obama Digital team. They optimised just about everything from web pages to emails. To give you an idea: They conducted 500 A/B tests on their web pages in a 20 month period. Resource: • https://blog.optimizely.com/2013/01/15/how-obamas-campaign-team-sourced-ab-tests-from-user-feedback/ • https://www.wired.com/2012/04/ff_abtesting/ • https://www.optimizely.com/customers/obama2012/ • https://www.youtube.com/watch?v=71bH8z6iqSc • http://kylerush.net/blog/optimization-at-the-obama-campaign-ab-testing/
- 14. How Spotify is doing it Spotify strongly believes, data makes you customer focused. Resource: • https://www.slideshare.net/alisarrafi3/ab-testing-at-spotify • https://www.slideshare.net/dj4b1n/ab-testing-pitfalls-and-lessons-learned-at-spotify-31935130 • https://www.slideshare.net/alisarrafi3/being-a-data-driven-business
- 15. How to get started _
- 16. Example 1a: Test understanding/validating of product benefits Key Steps To set a benchmark, isolate two to three tariffs (leave design untouched) and set up a preference test. Recruit participants from Amazon Turk or in- house (existing and non-existing customers). Isolate key benefits and let participants sort them by priority. Use an online card-sorting tool. Show one tariff and let participants freely sort and annotate elements they do not understand. Effort & Time Low effort, 1.5h to 2.5h Tools needed optimalworkshop.com or Helio from Zurb Recruiting Per email, social media, etc. Small sample size >10
- 17. Example 1b: Test understanding/ interpretation of icons Key Steps Take screenshots of new/ambiguous icons and embed them in an online survey. Underneath each icon, allow people to interpret the meaning of it via a free form field (or selection of words). Share it first internally and if more responses are needed, share online with a larger group. Alternatively, kick-things off internally and just share icons on slack to get some early/fast responses. Use your friends and just WhatsApp them the test-item. Get creative. Effort & Time low effort, 0.5h to 1h Tools needed Survey-Monkey, Typeform, Google Docs, WhatsApp Recruiting Per email, social media, WhatsApp, etc. Small sample size >10
- 18. Example 2: Test Layout & UI elements Key Steps Create or show people an existing screenshot. Share it with your audience and let them annotate/ comment on specific elements. Start with questions such as “What would you expect to happen when you click on …” -or- “Please tell us what … does” -or- “Can you show us where you would click to do/go to …” and so on. Great for initial A/B testing. Effort & Time Depending on complexity of screen, low- to mid- effort, 2h to 8h Tools needed Helio from Zurb, SurveyMonkey Recruiting Per email, social media, WhatsApp, etc. Small sample size >25 to 50
- 19. Example 3: Test functionality of features, modules or pages Key Steps Create a task, e.g. “Please find tariff X” -or- keep it more open and set up an easy-to-grasp scenario. Next, build a clickable prototype (low/high fidelity), so participants can run through the task you gave them. Observe or automate recording of session(s). Effort & Time High effort, 1 to 2 days. Tools needed Prototyping tool (Sketch+Invision), screen-recording setup (Usertesting.com or Lookback.io) Recruiting Per email, social media, WhatsApp, AmazonTurk, etc. Small sample size >25 to 50
- 20. Example 4: Test functionality & page performance Key Steps Focus on a key functionality or section/flow of your app/website. Choose an experiment and set-up event tracking and visual analysis tools. This type of testing requires modifying code and the ability to deploy the master within hours/ minutes. It is great for getting a large sample size >1000. Effort & Time High effort, multiple days to prep. Less, once automated testing is setup Tools needed Mixpanel or GA, Hotjar, Mousflow, etc. depending on test-type, Cohort Analysis Recruiting Per email, social media, WhatsApp, AmazonTurk, etc. Large sample size >1000
- 22. • Test early and often small, but relevant bits (early feedback is good feedback) • Automate most steps to speed things up • Share learnings frequently
- 26. Cool tools 1. https://zurb.com/helio/ 2. https://unbounce.com/ 3. https://www.usertesting.com/ 4. https://www.invisionapp.com/ in combination with https://lookback.io/ 5. https://www.hotjar.com/ or https://www.crazyegg.com/ 6. https://www.userzoom.com/ 7. https://validately.com/ 8. https://www.whatusersdo.com/ 9. https://fivesecondtest.com/ 10.https://www.optimizely.com/ 11.More tools: http://remoteresear.ch/tools/
- 27. MANY THANKS. You can hire me. Send an email ›