Query or Not to Query? Using Apache Spark Metrics to Highlight Potentially Problematic Queries
0 likes158 views
The document discusses Unravel's capabilities in optimizing data operations through predictive analytics for query performance and resource scheduling. It highlights features such as debugging code, maintaining service-level agreements, and providing operational insights to enhance development efficiency. Additionally, it outlines a schedule for a demo session and encourages user feedback.
Query or Not to Query? Using Apache Spark Metrics to Highlight Potentially Problematic Queries
- 1. Query or Not to Query Ask Unravel Prajakta Kalmegh, Principal Engineer Yusaku Sako, Head of Data Science
- 2. Data Science@ Prajakta Kalmegh ▪ [email protected] ▪ https://www.linkedin.com/in/pkalmegh/ Yusaku Sako ▪ [email protected] ▪ https://www.linkedin.com/in/yusaku-sako/
- 3. Experienced Team Strong Market Validation A Microsoft M12 Company Broad Technology and Cloud Platform Coverage Our Pedigree
- 5. Select right tech for the app, infrastructure, environment and cluster Debug code and pipelines, predict issues and assist in optimizing apps Check for app correctness and resource efficiency Container sizing, Scheduling and Cluster selection Tuning apps and eliminating rogue apps Proactive and automated actions to maintain SLAs CONTINUOUS INTEGRATION C O N TIN U O U S O PTIM IZA TIO N
- 9. Meet John 8:00 am 11:00 am 11:30 am 4:30 pm Still not working?
- 10. Why is this hard? What if I tweak my query? Did my resource requirements change? Is my data skewed on xxx? Is the cluster bottlenecked?
- 11. Unravel brings data-driven insights as you code
- 12. The notebook Demo 2Q | | !2Q
- 13. What Unravel exploits? ▪ Users often issue similar queries ▪ Same challenges faced ▪ Same mistakes repeated ▪ By the same user, by other users Holistic view of what worked and what did not
- 15. When is a good time to schedule? These are my resource requirements, should I schedule now or later? This report needs to be ready before Monday morning, my start time is flexible I have a new workload to schedule, what is a good time to start it?
- 16. Finding the missing piece Is the cluster slow (er) ? Has the query needs changed over time? Is it both?
- 17. Unravel uses predictive analytics to time it right
- 18. The timeit Demo while (!best) { # find better }
- 19. What Unravel exploits? ▪ Cluster utilization timeseries ▪ Query execution variability data ▪ Similar resource profiles and consumption history Holistic view of when it worked and when it didn’t
- 21. DELAYED Delayed: Increase in Input Data Size Detected
- 22. Delayed: Increase in Input Data Size Detected DELAYED
- 23. Architecture UNRAVELDAEMONS UnravelAPI Query Historical Executions Cluster State Indicators Query Quality Predictor Best Slot Finder AskUnravel
- 24. Already Scheduled Scheduled Query User Adhoc Query New Schedule Predict query issues Predict cluster issues Predict delays Submit Query ✓ SLA-bound ✓ Responsive wantstosubmit/track askunravel Submit Query Better Slots are xxx On Track Delayed by xxx Use-Cases Hold and submit lateraskunravel Better Slots are xxx Operational Insights Optimize Deployment Speedup Development
- 25. Recap: Detect Issues as early as possible 8:00 am 8:30 am Enjoy your day!
- 26. Thank you for watching! Signup for a free trial today https://bit.ly/3mo2ira
- 27. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.