Building a Culture of Experimentation at HP
4 likes•1,483 views
HP Inc. has developed a culture of experimentation through a structured global Center of Excellence, focusing on technology and governance to enhance digital experiences and sales. The company’s experimentation journey emphasizes data-driven decision-making, leading to significant increases in revenue and campaign volume. Key strategies include ideation sessions, cross-functional collaboration, and the application of scientific methods to optimize customer interactions.
Building a Culture of Experimentation at HP
- 1. 1 Building A Culture Of Experimentation At HP Inc. Hear how HP has built a global center of excellence, structured their experimentation teams, and focused on technology, governance, and strategy on their way to building a culture of experimentation.
- 2. 2 Introductions Neville Davey Global Experimentation Lead David Carlile Customer Success Manager Anmeen Leong Americas Experimentation Lead
- 3. 3 Housekeeping I. We are recording this webinar II. You’ll receive the recording and slides III. We’ll answer all questions at the end
- 5. 5 What do we do at Optimizely?
- 6. Revenue, share of wallet, funnel conversion, risk mitigation, ops efficiency What We Do Next gen “Test and Learn” platform for enterprise- wide digital experience optimization Over 1B impressions daily Replaces Digital Guesswork With Evidence-based Optimization Digital Experimentation SaaS Platform Apply the scientific method to “at-scale” business decision making. 20X increase in Yield
- 7. What We Enable 1500+ +51% +19% +28% Improved subscription acquisition Users Scale and agility of experiments; users Improved Driver Acquisition increase in cross-sell performance
- 8. Spent with Digital Media 19 hr week 2018 Digital-Centric Customer Spent with Digital Media 41 $396m Mobile commerce Mobile commerce $1 8t. 2008 Digital-Convenient Customer hr week Your customer has fundamentally changed how they engage
- 9. Wealthfront managed $1B in assets 3x faster than Schwab RETAIL Amazon makes up 50 percent of the U.S. e-commerce market. (Walmart 4%) TRAVEL Airbnb has more room listings than top 5 hotel brands combined HIGH TECH Teams using Slack see an average reduction in email volume of 48.6% MEDIA 33 million people cut their cords in 2018 The Digital Experience Economy Winning in digital is a matter of life or death in many industries How do win? FINANCE
- 10. Experimentation: Testing variations of digital experiences with real users and deploying what wins +21% -5%
- 11. Frontend UI Backend Business Logic & Data The anatomy of an experience i.e. navigation, search location & visual treatment Copy Images & Colors Layout Search algorithms Personalize content based on previous behavior Recommendations Make your headlines more personal
- 12. Web / SPA Trusted results Adoption & scale Integration with key systems Mobile OTT Governance Process change, implementation support Security & compliance Frontend UI Optimizing these experiences is extremely challenging Backend Business Logic & Data Personalization IOT / Voice Marketing Product & Engineering Support
- 13. Device coverage Complete experience coverage Open platform Trusted results quickly Consumer-grade usability Security & compliance Program Management Strategic & tactical partnership Build Digital Products Market & Sell Service & Retain What you need Marketing Product & Engineering Support The Optimizely Solution Consumer grade usability for adoption & velocity Program Management Purpose- Built Statistics Engine & Machine Learning Open Platform Data & Integrations Security & Compliance OTT Voice/IOT Mobile Web / SPA Frontend and Server-side Experimentation & Personalization Best in class Services & Success teams SUCCESS ACCELERATIONEXPERIENCE COVERAGEDEVICE COVERAGE TRUSTED RESULTS ENTERPRISE SCALE
- 14. We’re Proud to Work With Great Global Enterprises 24 of the Fortune 100 have chosen Optimizely to drive their digital experience
- 16. 16 CULTURE OF EXPERIMENTATION WINNING Setting The Stage at HP Where We Were Two Years Ago Where We Are Today Where We Want To Be
- 17. To improve Omnichannel experiences that delight our customers and drive sales through data-driven decision making. HP’s Experimentation Vision
- 18. 18 BUSINESS VALUE VELOCITY/VOLUME LEVEL 1 Executional Start LEVEL 2 Foundational Growth LEVEL 3 Cross-functional Advancement LEVEL 4 Operational Excellence LEVEL 5 Culture of Experimentation 2015 US B2C Store Quick Wins 2016-17 B2C Geo Expansion B2B, Marketing & Support Sites CoE Program started 2018 APJ & EMEA Expansion Segment based personalization HP’s EXPERIMENTATION JOURNEY
- 19. FY18 Major Milestones Formalized Testing Program (Ideation sessions, workflow, reporting) Created Experimentation CoE (Best practice sharing, promote test culture) Expanded to All Regions (1 to 3 regions, 10+ countries) More Stakeholders (3 Region Stores, HP.com, B2B, Support, Instant Ink) Increased Campaign Volume (2x increase Y/Y) Increased Value to HP (>3x revenue increase Y/Y) FY19 Target: 3x Campaign Volume
- 20. I. Our emphasis on a strong end-to-end program management foundation. II. How we’ve interlocked our technologies to give us insights and the edge on our Experiments. III. How we structured our Center Of Excellence, and why it's working for us at a global scale to increase campaigns by 3x. IV. Examples of how experimentation has brought value to HP. What we want to share with you today:
- 21. 21 Individual Champion 2015 US Level 1 Challenges • Low Activity • Lack of Knowledge / Value • Credibility
- 22. 22 Individual Champion 2015 2018 COE US Americas Asia Pacific Europe B2B Enterprise Customer Support Instant Ink Full Service (Testing Council Model) Enable (COE Model) Level 1 Level 3 ?
- 23. 23 Individual Champion 2015 2018 COE US Americas Asia Pacific Europe B2B Enterprise Customer Support Instant Ink Full Service (Testing Council Model) Enable (COE Model) Program Management ü Strategy ü Communications ü Resources & Value Technology & Data ü Tech Stack ü Data ü Code Quality Level 1 Level 3 Worldwide Growth ü Program Structure ü Service Model ü Team Structure
- 25. 25 Individual Champion 2015 2018 COE US Americas Asia Pacific Europe B2B Enterprise Customer Support Instant Ink Full Service (Testing Council Model) Enable (COE Model) Program Management ü Strategy ü Communications ü Resources & Value Technology & Data ü Tech Stack ü Data ü Code Quality Level 1 Level 3 Worldwide Growth ü Program Structure ü Service Model ü Team Structure
- 26. 26 Strategy • Based on business goals • Ideate and prioritize with stakeholders • Informed by analytics and customer feedback KPIs • Vary by stakeholder • eCommerce: CR, RPV • Call/Chat Center: Call volume • Marketing: Engagement, CTR
- 27. 27 A/B test was aimed to increase global nav referrals and measure how increasing traffic to the site would influence other key performance indicators. Other referrers Lead-gen start rate Lead-gen starts Global nav referrals 3D page visits Lead-gen start rate Completion % per step # of steps Typed / bookmarked 3D Printer Revenue Leads generatedLead quality Test KPI BU Strategic Goal Test Change
- 28. Prioritization Create a meeting structure for US Store Covers test cycle across multiple teams 28 Print Meeting MEETINGS Ideation Results Sharing Alignment Meeting PC Meeting Print Category Combined Category PC Category Print Meeting PC MeetingAlignment Meeting Alignment Meeting Reporting Meeting Create tickets in JIRA 4x Meetings 1x Print Weekly 1x PC Weekly 1x Alignment Weekly 1x Reporting Weekly & Alignment Create tickets in JIRACreate tickets in JIRA
- 29. List of Tools used to coordinate the program 29 Ideation Collect and gather test ideas from all over the organization Prioritization / Roadmap Test Plans Results Sharing Prioritize based on impact, level of effort, and initiatives Document and explain the key information in for your experiment Share test results and insights broadly with HP organization Optimizely Program Management
- 30. Experimentation Hub (intranet site) for Reporting & Resources 30
- 31. Email Newsletter & Yammer for Communications 31
- 32. Evangelizing to execs and broader company: Program Dashboard (OptimizelyPM, Tableau), Internal Webinar 32
- 33. 33 • Strategy based on business goals • Communicate and align with stakeholder • Communicate broadly, show value Bottom Line
- 34. 34 Technology & Data Tech Stack Data Code Quality
- 35. 35 Individual Champion 2015 2018 COE US Americas Asia Pacific Europe B2B Enterprise Customer Support Instant Ink Full Service (Testing Council Model) Enable (COE Model) Program Management ü Strategy ü Communications ü Resources & Value Technology & Data ü Tech Stack ü Data ü Code Quality Level 1 Level 3 Worldwide Growth ü Program Structure ü Service Model ü Team Structure
- 36. Tech stack enables quality code and data 36 Development Task Automation Workflow Optimizely Custom Snippet (cross-domain/project integration) Optimizely for Dynamic Websites (single page application) Quantitative Data Qualitative Data
- 37. Analytics Tools in Action (integrated with Optimizely) 37 Breakdown experiment sales by product category User feedback and pain points to inform experimentation roadmap User behavior in experiment
- 38. 38 Development Task Automation Workflow Mohammad Alabandi Lead Experimentation Developer HP Coming Soon to Features • Build outside Optimizely using code editor of choice • Lints and optimizes javascript files, browser compatibility • Direct publish to Optimizely (via API) Benefits ü Velocity: Development speed increased 40% ü Quality: Code errors reduced 20% ü Open source: Shared across HP stakeholders
- 39. 39 Worldwide Growth Program Structure Service Model Team Structure
- 40. 40 Individual Champion 2015 2018 COE US Americas Asia Pacific Europe B2B Enterprise Customer Support Instant Ink Full Service (Testing Council Model) Enable (COE Model) Program Management ü Strategy ü Communications ü Resources & Value Technology & Data ü Tech Stack ü Data ü Code Quality Level 1 Level 3 Worldwide Growth ü Program Structure ü Service Model ü Team Structure
- 41. 41 Individual Single team and single site experience Small focus, simple tests, build momentum Program Manager with access to Dev, Designer, Etc. Center Of Excellence Central support team for business units (own individual experiences) Enabling and supporting individual teams, and sharing best practices Central expert team (Including PM, Dev, Designer, Analyst, Etc.) Testing Council Shared ownership of overall experiences and prioritization Broader digital initiatives requiring coordination Testing Council for Analytics, UX, Etc. Definition Characteristics Team Common Types of Program Structure
- 42. 42 Testing Council Individual Center Of Excellence Common Types of Program Structure Decentralized Centralized One Many Culture/Ownership Number of Teams
- 43. 43 Testing Council Individual Center Of Excellence Common Types of Program Structure Decentralized Centralized One Many Culture/Ownership Number of Teams
- 44. 44 Testing Council Individual Center Of Excellence Common Types of Program Structure Decentralized Centralized One Many Culture/Ownership Number of Teams
- 45. FULL SERVICE Per Quarter fee ENABLE ONLY Per Quarter fee FREE TRIAL One quarter AB Testing platform Onboarding / training Monthly Unique Visitor Allocation Out-of-the-box reporting Test bandwidth Advice throughout testing process, best practices Full definition & set-up (Web / Mobile dev, UX, Analytics) Launch test (with adjustments) Full analysis and custom reporting Insights, recommendations & iteration advice Experiment revision(s) Program Governance 1 to 1 Sessions Community Sessions (Office Hours) Playbook on launching campaigns/Wiki Site Access 1m (Funded) Unlimited (Paid) Unlimited (Paid) Up to 5 tests Unlimited Unlimited - - - - - - - - - - - Most Popular Package HP Service Plans: Stakeholders pick what’s best for them COE Model Testing Council Model
- 46. 46 Experimentation = Value for HP Revenue Growth Reduce Customer Interaction Cost Enable Operational Excellence Risk Mitigation
- 47. 47 DIGITAL EXPERIENCE OPTIMIZATION Mitigate RiskEnable Operational Excellence Deliver Measurable Revenue Growth Reduce Customer Interaction Costs
- 48. 48 DIGITAL EXPERIENCE OPTIMIZATION Deliver Measurable Revenue Growth
- 49. Problem: Gaming customers not aware of gaming offers 49 Original (Standard Ribbon) Variation (Personalized Ribbon) Results • +28% RPV for Gaming PCs (+13% RPV overall). • Explore other customer segments for simple personalization. 1 1 “Value Prop” storewide ribbon Sale Page / Weekly Deals Page “Value Prop” storewide ribbon Omen Sale Page • Value Prop ribbon has personalized (gaming) sale messaging • CTA goes to Omen Sale Page • Value Prop ribbon has standard sale messaging • CTA goes to standard sale page (sale / weekly deals)1 1 Action: Personalize the storewide ribbon messaging and landing page for “gaming inclined” visitors.
- 50. 50 DIGITAL EXPERIENCE OPTIMIZATION Reduce Customer Interaction Costs
- 51. Problem: Customers are calling the sales phone number for support calls 51 Original (“Order Status” Text) Variation (“Sales” Text) Results • +35% sales calls, -86% order status calls. • Customers still able to check order status online via web link. • Order Status link in position3. • Phone number now in position1, including the “Sales” text. • Order Status link in position1. Action: Move “order status” link to less prominent position, add “sales” text to clarify phone number purpose 1 2 1 1 2 1
- 53. Problem: Companywide “HPID” initiative may increase checkout friction 53 Results • Net effect: No harm to sales. • High-ticket item shoppers less impacted than low-ticket item shoppers. Original (Standard Checkout) Variation 1 (HPID Checkout) • Logging in from Cart page sends user directly to Checkout page • Logging in from Cart page, user will see an intermediate HPID password page, before proceeding to Checkout page Action: Added additional step/page (HPID) in checkout flow prior to formal rollout, to gauge impact on conversion & CX
- 54. 54 DIGITAL EXPERIENCE OPTIMIZATION Enable Operational Excellence
- 55. Problem: Black Friday Sale Page not optimized for speed and CX 55 Original (Classic Webpage) Variation (Single Page Application) Results • +15% Conversion. Single Page Application domain rolled out for Black Friday and Cyber Monday sale pages. • Contributed to double-digit Y/Y sale growth. • HP Pre-Black-Friday Sale Page UI (identical) • Faster page load times (in theory). • Single page application, server side rendering. • HP Pre-Black-Friday Sale Page UI Action: Developed new single page application (newer technology) with faster page speed
- 56. 56 In Summary…
- 57. 57 Individual Champion 2015 2018 COE US Americas Asia Pacific Europe B2B Enterprise Customer Support Instant Ink Full Service (Testing Council Model) Enable (COE Model) Program Management ü Strategy ü Communications ü Resources & Value Technology & Data ü Tech Stack ü Data ü Code Quality Level 1 Level 3 Worldwide Growth ü Program Structure ü Service Model ü Team Structure
- 58. Looking Ahead 58 Complexity vs Velocity • How to balance creating more complex tests vs more volume of tests? • Bottom line: what mix will drive the most value? Personalization • Current: broad customer segmentation, account-based segmentation • Near future: increase use of 3P data, more granular personalization BU Expansion • Reach more business units, ramp existing stakeholders Server-Side Testing • Evaluate Optimizely Full Stack platform
- 59. 59 Q & A If you questions for HP please submit in the chat box now!