Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
3 likes3,091 views
This presentation introduces how we design and implement a real-time processing platform using latest Spark Structured Streaming framework to intelligently transform the production lines in the manufacturing industry. In the traditional production line there are a variety of isolated structured, semi-structured and unstructured data, such as sensor data, machine screen output, log output, database records etc. There are two main data scenarios: 1) Picture and video data with low frequency but a large amount; 2) Continuous data with high frequency. They are not a large amount of data per unit. However the total amount of them is very large, such as vibration data used to detect the quality of the equipment. These data have the characteristics of streaming data: real-time, volatile, burst, disorder and infinity. Making effective real-time decisions to retrieve values from these data is critical to smart manufacturing. The latest Spark Structured Streaming framework greatly lowers the bar for building highly scalable and fault-tolerant streaming applications. Thanks to the Spark we are able to build a low-latency, high-throughput and reliable operation system involving data acquisition, transmission, analysis and storage. The actual user case proved that the system meets the needs of real-time decision-making. The system greatly enhance the production process of predictive fault repair and production line material tracking efficiency, and can reduce about half of the labor force for the production lines.
![[Spark Summit EU 2017] Apache spark streaming + kafka 0.10 an integration story](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkstreamingkafka0-171026072955-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu (20)
Ad
More from Spark Summit (20)
Ad
Recently uploaded (20)
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
- 1. #EUstr1 Intel Big Data Engineering Team Spark Structured Streaming Helps Smart Manufacturing Xiaochang Wu [email protected] Hao Cheng [email protected]
- 2. Who am I? • Currently working at Intel Big Data Engineering team based in Shanghai, China • Joined Intel from 2006, focus on performance optimization for CPU / GPU / System / App for ~10 years • Delivering the best Spark performance on Intel platforms • Building reference big data platform for Intel customers 2#EUstr1
- 3. Agenda • Motivation • Architecture and Features Overview • Adopting Spark Structured Streaming • Improvement Proposals 3#EUstr1
- 4. Motivation • Building Spark-centric IOT Data Platform for PCB Manufacturing – Data collection, processing, analytics, visualization and alerting – Short-term goal: Enhance predictive fault repair and material tracking efficiency • Take advantage of latest Structured Streaming – Streaming ETL – Stateless and Stateful processing • Learn from real-world problems 4#EUstr1
- 5. Overview • Transform the PCB surface mount lines • SPI: Solder Paste Inspection • AOI: Automated Optical Inspection 5#EUstr1 Solder Paste Printing SPI Component Placement Pre-Reflow AOI Reflow Soldering Post- Reflow AOI
- 6. Data Platform Features • Device Management – Register and manage devices – RPC commands to access device attributes • Data Collection – Collect and store device data in reliable way • Data Processing & Analytics – Domain-specific scripts for both batch and real-time processing – Support using Machine Learning algorithms for advanced analytics • Rule Engine & Actions – Process incoming data with flexible user-defined rules. – Trigger alarms or send commands to devices using rules. • Data Visualization – Customizable user-friendly UI & Dashboards – Generate interactive data reports to fulfil business needs • Multi-tenancy, Scale-out, Fault- tolerance & Security 6#EUstr1
- 7. Data Platform Architecture 7#EUstr1 *Other names and brands may be claimed as the property of others.
- 8. Data Characteristics • Data Sources – Manufacturing Execution System (MES) – Sensor, Camera, Log – OCR for machine screen output • Structured, semi-structured and unstructured data – Low frequency but large amount • E.g. Picture and video – High frequency continuous data • Not large amount per unit. Total amount is very large • E.g. vibration sensor data used to detect the quality of the equipment. 8#EUstr1
- 9. Spark Structured Streaming • Built on the Spark SQL engine – Express streaming the same way as batch – Dataset/DataFrame API in Scala, Java, Python or R • Incrementally and continuously updating the final result – Handling Event-time and Late Data – Delivering end-to-end exactly-once fault tolerance semantics – Output modes: Append, Complete, Update mode • Support Stateless and Stateful operations 9put your #assignedhashtag here by setting the footer in view-header/footer Fast, scalable, fault-tolerant, end-to-end exactly-once
- 10. Streaming ETL • Data are produced from sources as JSON objects, transferred with MQTT and saved to Kafka • JSON objects are complex and have nested structures • Use Spark SQL functions to extract – selectExpr, get_json_object, cast, alias 10#EUstr1
- 11. Stateless and Stateful Operations • Stateless – Projection, Filter • Stateful (Implicit) – Aggregation – Time window – Join • Advanced arbitrary stateful operations – mapGroupsWithState / flatMapGroupsWithState 11#EUstr1
- 12. Structured Streaming Considerations • Output mode: append, update, complete • Handling late data – Method 1: use time window and automatic watermark – Method 2: use manual watermark • Post-processing – Debug: console sink / memory sink – File sink / Foreach sink – Collect to driver side and process 12#EUstr1
- 13. Case 1: Find out offset AVG and STDDEV in Solder Paste Inspection (SPI) 13#EUstr1
- 14. Case 2: Find out machine off time from event logs (1) • Track sessions from streams of events and find out machine off time – mapGroupsWithState / flatMapGroupsWithState – Off time = Duration from status “off” to next “on” 14#EUstr1
- 15. Case 2: Find out machine off time from event logs (2) • Event: define data types for events • SessionInfo: Session information as state • SessionUpdate: Update information returned as result 15#EUstr1
- 16. 16#EUstr1
- 17. Case 2: Find out machine off time from event logs (3) • Event time ordering • Every subsequent event depends on previous event • Required to sort events based on timestamp due to logical dependence • Handling late events • Late data will break previous calculation result, forced to maintain all events and recalculate • Solution: manual watermarking for late data, remove old events if timeout, incremental calculation 17#EUstr1
- 18. Improvement Proposals • Flexible window and trigger operation • Complex Event Processing (CEP) APIs – Stream Join – Session Tracking – Pattern – Alert • Streaming Domain Specific Language (DSL) – SQL-like language for Streams • We are working on them! 18#EUstr1
- 19. See Also from Intel team • Today 14:00 @ AUDITORIUM An Adaptive Execution Engine For Apache Spark SQL by Carson Wang • Tomorrow 11:00 @ LIFFEY HALL FPGA-Based Acceleration Architecture for Spark SQL by Qi Xie 19#EUstr1
- 20. Legal Notices and Disclaimers • You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. • No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. • Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. 20#EUstr1
- 21. References • Surface mount process: http://www.surfacemountprocess.com/ • Structured Streaming Programming Guide: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html • Five Spark SQL Utility Functions to Extract and Explore Complex Data Types: https://databricks.com/blog/2017/06/13/five-spark-sql-utility-functions-extract-explore- complex-data-types.html • Spark Structured Sessionization example: https://github.com/apache/spark/blob/v2.2.0/examples/src/main/scala/org/apache/spa rk/examples/sql/streaming/StructuredSessionization.scala 21#EUstr1