Apache Spark Performance: Past, Future and Present
4 likes2,094 views
This document discusses improving performance instrumentation in Apache Spark. It summarizes that existing instrumentation focuses on blocked times in the main task thread, but opportunities exist to better instrument read/write times and machine-level resource utilization. The author proposes combining per-task I/O metrics with machine utilization data to provide complete metrics about time spent using each resource on a per-task basis, improving performance clarity for users. More details are available at the listed website.
Ad
More Related Content
What's hot (20)
Similar to Apache Spark Performance: Past, Future and Present (20)
Ad
More from Databricks (20)
Ad
Recently uploaded (20)
![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
![Pixologic ZBrush Crack Plus Activation Key [Latest 2025] New Version](https://cdn.slidesharecdn.com/ss_thumbnails/fashionevolution2-250322112409-f76abaa7-250428124909-b51264ff-250504160528-fc2bb1c5-thumbnail.jpg?width=560&fit=bounds)
Apache Spark Performance: Past, Future and Present
- 1. Spark Performance Past, Future, and Present Kay Ousterhout Joint work with Christopher Canel, Ryan Rasti, Sylvia Ratnasamy, Scott Shenker, Byung-Gon Chun
- 2. About Me Apache Spark PMC Member Recent PhD graduate from UC Berkeley Thesis work on performance of large-scale data analytics Co-founder at Kelda (kelda.io)
- 3. How can I make this faster?
- 4. How can I make this faster?
- 5. Should I use a different cloud instance type?
- 6. Should I trade more CPU for less I/O by using better compression?
- 7. How can I make this faster? ???
- 8. How can I make this faster? ???
- 9. How can I make this faster? ???
- 10. Major performance improvements possible via tuning, configuration …if only you knew which knobs to turn
- 11. Past: Performance instrumentation in Spark Future: New architecture that provides performance clarity Present: Improving Spark’s performance instrumentation This talk
- 12. spark.textFile(“hdfs://…”) .flatMap(lambda l: l.split(“ “)) .map(lambda w: (w, 1)) .reduceByKey(lambda a, b: a + b) .saveAsTextFile(“hdfs://…”) Example Spark Job Split input file into words and emit count of 1 for each Word Count:
- 13. Example Spark Job Split input file into words and emit count of 1 for each Word Count: For each word, combine the counts, and save the output spark.textFile(“hdfs://…”) .flatMap(lambda l: l.split(“ “)) .map(lambda w: (w, 1)) .reduceByKey(lambda a, b: a + b) .saveAsTextFile(“hdfs://…”)
- 14. spark.textFile(“hdfs://…”) .flatMap(lambda l: l.split(“ “)) .map(lambda w: (w, 1)) Map Stage: Split input file into words and emit count of 1 for each Reduce Stage: For each word, combine the counts, and save the output Spark Word Count Job: .reduceByKey(lambda a, b: a + b) .saveAsTextFile(“hdfs://…”) … Worker 1 Worker n Tasks … Worker 1 Worker n
- 15. Spark Word Count Job: Reduce Stage: For each word, combine the counts, and save the output .reduceByKey(lambda a, b: a + b) .saveAsTextFile(“hdfs://…”) … Worker 1 Worker n
- 16. compute network time (1) Request a few shuffle blocks disk (5) Continue fetching remote data : time to handle one shuffle block (2) Process local data What happens in a reduce task? (4) Process data fetched remotely (3) Write output to disk
- 17. compute network time disk : time to handle one shuffle block What happens in a reduce task? Bottlenecked on network and disk Bottlenecked on network Bottlenecked on CPU
- 18. compute network time disk : time to handle one shuffle block What happens in a reduce task? Bottlenecked on network and disk Bottlenecked on network Bottlenecked on CPU
- 19. compute network time disk What instrumentation exists today? Instrumentation centered on single, main task thread : shuffle read blocked time : executor computing time (!)
- 20. actual What instrumentation exists today? timeline version
- 21. What instrumentation exists today? Instrumentation centered on single, main task thread Shuffle read and shuffle write blocked time Input read and output write blocked time not instrumented Possible to add!
- 22. compute disk Instrumenting read and write time Process shuffle block This is a lie compute Reality: Spark processes and then writes one record at a time Most writes get buffered Occasionally the buffer is flushed
- 23. compute Spark processes and then writes one record at a time Most writes get buffered Occasionally the buffer is flushed Challenges with reality: Record-level instrumentation is too high overhead Spark doesn’t know when buffers get flushed (HDFS does!)
- 24. Tasks use fine-grained pipelining to parallelize resources Instrumented times are blocked times only (task is doing other things in background) Opportunities to improve instrumentation
- 25. Past: Performance instrumentation in Spark Future: New architecture that provides performance clarity Present: Improving Spark’s performance instrumentation This talk
- 26. Task 1 Task 2 Task 5 Task 3 Task 4 Task 7 Task 6 Task 8 time 4 concurrent tasks on a worker
- 27. Task 1 Task 2 Task 5 Task 3 Task 4 Task 7 Task 6 Task 8 time Concurrent tasks may contend for the same resource (e.g., network)
- 28. What’s the bottleneck? Task 1 Task 2 Task 5 Task 3 Task 4 Task 7 Task 6 Task 8 Time t: different tasks may be bottlenecked on different resources Single task may be bottlenecked on different resources at different times
- 29. Task 1 Task 2 Task 5 Task 3 Task 4 Task 7 Task 6 Task 8 How much faster would my job be with 2x disk throughput? How would runtimes for these disk writes change? How would that change timing of (and contention for) other resources?
- 30. Today: tasks use pipelining to parallelize multiple resources Proposal: build systems using monotasks that each consume just one resource
- 31. Monotasks: Each task uses one resource Network monotask Disk monotask Compute monotask Today’s task: Monotasks don’t start until all dependencies complete Task 1 Network read CPU Disk write
- 32. Dedicated schedulers control contention Network scheduler CPU scheduler: 1 monotask / core Disk drive scheduler: 1 monotask / disk Monotasks for one of today’s tasks:
- 33. Spark today: Tasks have non- uniform resource use 4 multi-resource tasks run concurrently Single-resource monotasks scheduled by per-resource schedulers Monotasks: API-compatible, performance parity with Spark Performance telemetry trivial!
- 34. How much faster would job run if... 4x more machines Input stored in-memory No disk read No CPU time to deserialize Flash drives instead of disks Faster shuffle read/write time 10x improvement predicted with at most 23% error � ��� ��� ��� ��� ���� ���� ���� ���� ��� ����� �������� �� ����� �� ����� ���������� �������� �������� ������� �� ��������� ���� ���� ������ ��������� ��� ������� ��� ��������� ���� ���� ������ ������ ��� ������� ��� ��������� ���� ���� ������
- 35. Monotasks: Break jobs into single-resource tasks Using single-resource monotasks provides clarity without sacrificing performance Massive change to Spark internals (>20K lines of code)
- 36. Past: Performance instrumentation in Spark Future: New architecture that provides performance clarity Present: Improving Spark’s performance instrumentation This talk
- 37. Spark today: Task resource use changes at fine time granularity 4 multi-resource tasks run concurrently Monotasks: Single-resource tasks lead to complete, trivial performance metrics Can we get monotask-like per-resource metrics for each task today?
- 38. compute network time disk Can we get per-task resource use? Measure machine resource utilization?
- 39. Task 1 Task 2 Task 5 Task 3 Task 4 Task 7 Task 6 Task 8 time 4 concurrent tasks on a worker
- 40. Task 1 Task 2 Task 5 Task 3 Task 4 Task 7 Task 6 Task 8 time Concurrent tasks may contend for the same resource (e.g., network) Contention controlled by lower layers (e.g., operating system)
- 41. compute network time disk Can we get per-task resource use? Machine utilization includes other tasks Can’t directly measure per-task I/O: in background, mixed with other tasks
- 42. compute network time disk Can we get per-task resource use? Existing metrics: total data read How long did it take? Use machine utilization metrics to get bandwidth!
- 43. Existing per-task I/O counters (e.g., shuffle bytes read) + Machine-level utilization (and bandwidth) metrics = Complete metrics about time spent using each resource
- 44. Goal: provide performance clarity Only way to improve performance is to know what to speed up Why do we care about performance clarity? Typical performance eval: group of experts Practical performance: 1 novice
- 45. Goal: provide performance clarity Only way to improve performance is to know what to speed up Some instrumentation exists already Focuses on blocked times in the main task thread Many opportunities to improve instrumentation (1) Add read/write instrumentation to lower level (e.g., HDFS) (2) Add machine-level utilization info (3) Calculate per-resource time More details at kayousterhout.org