AI/ML Infra Meetup | Scaling Experimentation Platform in Digital Marketplaces: Architecture, Implementation & Lessons Learned
0 likes728 views
AI/ML Infra Meetup Aug. 29, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Koundinya Pidaparthi (VP of Analytics @ Poshmark) Scaling experimentation in digital marketplaces is crucial for driving growth and enhancing user experiences. However, varied methodologies and a lack of experiment governance can hinder the impact of experimentation leading to inconsistent decision-making, inefficiencies, and missed opportunities for innovation. At Poshmark, we developed a homegrown experimentation platform, Lightspeed, that allowed us to make reliable and confident reads on product changes, which led to a 10x growth in experiment velocity and positive business outcomes along the way. This session will provide a deep dive into the best practices and lessons learned from successful implementations of large-scale experiments. We will explore the importance of experimentation, overcome scalability challenges, and gain insights into the frameworks and technologies that enable effective testing.
![[Webinar] Visa's Journey to a Culture of Experimentation](https://cdn.slidesharecdn.com/ss_thumbnails/optimizelywebinar-visasjourneytoacultureofexperimentation-180227210737-thumbnail.jpg?width=560&fit=bounds)
![[Webinar] Getting started with server-side testing - presented by WiderFunnel...](https://cdn.slidesharecdn.com/ss_thumbnails/serversidetestingwebinardeck-171213214850-thumbnail.jpg?width=560&fit=bounds)
![[WSO2Con USA 2018] Winning Strategy For Enterprise Integration to Empower Dig...](https://cdn.slidesharecdn.com/ss_thumbnails/1130kingpaul-180718055022-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
Similar to AI/ML Infra Meetup | Scaling Experimentation Platform in Digital Marketplaces: Architecture, Implementation & Lessons Learned (20)
More from Alluxio, Inc. (20)
Ad
Recently uploaded (20)
![Minitab 22 Full Crack Plus Product Key Free Download [Latest] 2025](https://cdn.slidesharecdn.com/ss_thumbnails/avtpastoralvisits20252026-250407114051-0c8f0fbf-250407123507-bf8f5b04-250426175635-00bc6197-thumbnail.jpg?width=560&fit=bounds)
Ad
AI/ML Infra Meetup | Scaling Experimentation Platform in Digital Marketplaces: Architecture, Implementation & Lessons Learned
- 1. Scaling Experimentation @ Poshmark Koundinya Pidaparthi, VP Analytics Aug 2024
- 2. Poshmark is a leading fashion resale marketplace powered by a vibrant, highly engaged community of buyers and sellers and real-time social experiences. Designed to make online selling fun, more social and easier than ever, Poshmark empowers its sellers to turn their closet into a thriving business and share their style with the world.
- 3. POSHMARK NETWORK & SCALE Since its founding in 2011, Poshmark has grown its community to over 130 million users and generated over $10 billion in GMV shop from over 10k brands and 90 categories
- 4. EXPERIMENTATION IS CORE TO DELIVER QUALITY PRODUCT Dynamic marketplace, rely on experimentation and data to build tools, measure impact and regression on metrics. ● Continuous Weekly Releases ● Varied Use cases ○ Code Migrations ○ New Feature Launches ○ App Redesign ○ Campaign Management ● 100s of metrics monitored for regression ● 200+ Internal Users of Experimentation Platform
- 5. EXPERIMENTATION PLATFORM CHALLENGES ● Lack of consistent and reliable frameworks can slow down decision making. ● High exposure can lead to too much risk. ● Low experiment concurrency can slow down product velocity. ● Analytics resources are often tied up with tedious workflows and repetitive analysis. ● Experiment operations and coordination can be time consuming. ● Lack of shared learnings can lead to more cold starts.
- 6. Our goal is to build a reliable experimentation platform to accelerate product innovation, ship high quality product, streamline experiment processes and save time.
- 7. 3 MAIN COMPONENTS OF OUR EXPERIMENTATION PLATFORM Analysis and Decision Framework Standardization Robustness Generalized Use Cases 1 Experiment Operations Centralized Code Repo Metrics Management Analysis Sharing Experiment Management Tools 2 Feature Flagging Lowered exposure Expanded Capacity Assignment Algorithm 3
- 8. Analysis and Decision Framework ● Experiment Design Guidelines ○ Clarify Goals ○ Standardized Metrics Nomenclature ■ North Star ■ Feature Metrics ■ Guardrail Metrics ○ Sizing ○ Standardized Stats Engine ● Standardized Operating Rhythm and Decision Frameworks ○ Readout Reviews ○ Pre defined success criteria ~ Cut Experimentation Cycle time, achieved 3X Lift in Experimentation Volume!
- 9. Streamlined Experiment Operations - Analytics Architecture ● Lightspeed Metrics ○ Standardized Business Logic ○ Python Stats Package ○ Pre Processed ETLs ● Knowledge Repo for collaborative reporting ○ Templated Notebooks ○ Converts Notebooks into user friendly readouts ○ Centralized Learnings ● AB Console ○ On Demand Experiment Scheduling ○ Improved visibility to Tests on Platform ~ 20% to 30% Efficiency Gains for Analysts
- 10. Feature Flagging Enhancements Segments ~ 10x increase in capacity to run concurrent experiments ● Created more layers and segments ○ Increased Capacity to run more concurrent experiments ○ Allow for smaller exposure tests, mitigate risk ○ Support expanded Use cases, Holdouts, Multivariate Tests ● Built assignment algorithm ○ On demand assignment ○ More confidence in randomization
- 11. Lessons Learned ● Actively listened to user feedback and built MVP to address key friction points which led to immediate value and adoption of the platform. ○ Standardization and consistency were key to build confidence. ○ Reducing friction increases adoption ● Open data culture, knowledge sharing and healthy debate was critical in building a culture of experimentation. ○ Focus on learnings, every AB test is a success regardless of outcome. ○ Continuous training and accountability are key to sustained success. ● Be open to new use cases for experimentation. ○ Campaign management, infrastructure migrations, holdouts. ● Be strategic about investments. Continuous investments not always a good ROI due to competing priorities and limited resources.
- 12. THANK YOU!