Real-time Processing

Tiêu chí 1: Nêu định nghĩa và ảnh hưởng của mô hình lên hiệu suất hệ thống và
yêu cầu về cơ sở hạ tầng?

Tiêu chí 2: Nêu ưu, nhược điểm của mô hình và ứng dụng thực tế.

Tiêu chí 3: Thuật toán hỗ trợ cho từng loại mô hình và vấn đề thuật toán đó giải
quyết?

Tiêu chí 4: Liệt kê những công cụ hỗ trợ triển khai vào thực tế và phân tích về các
công cụ đó.

I. Data Processing (giới thiệu đề tài lớn để dẫn vào nội dung chính) (Thái)
In today's digital age, data processing is an essential part of almost every industry. It
plays a crucial role in enabling organizations to extract valuable insights from their
data to gain a competitive advantage, improve operational efficiency, and provide
better customer service.

Data processing occurs when data is collected and translated into usable
information. Usually performed by a data scientist or team of data scientists, it is
important for data processing to be done correctly as not to negatively affect the end
product, or data output.

Data processing starts with data in its raw form and converts it into a more readable
format (graphs, documents, etc.), giving it the form and context necessary to be
interpreted by computers and utilized by employees throughout an organization.

Types of Data Processing:

● Batch processing (Xử lý dữ liệu theo lô)
● Streaming processing (Xử lý dữ liệu theo luồng)
● Real-time processing (Xử lý dữ liệu theo thời gian thực)

II. What Is Real-Time Processing? (Thành)

Real-time processing is a method that involves collecting, processing, and analyzing
data as it is generated, allowing for near-instantaneous insights. In a real-time data
processing system, rapidly changing input data is processed, and output is provided
almost instantaneously, enabling quick response to changes. This approach is
applied when immediate processing is necessary, commonly found in structures
with computer control.

Real-time processing involves continual input processing of data from various

sources with very short latency, ensuring a constant flow of information. Unlike
batch processing, real-time processing occurs almost immediately without pauses
or waiting, offering advantages in responsiveness and agility.
III. Advantages And Of Real-Time Data Processing (Đồng)
Immediate Decision-Making & Responsive Actions

The major advantage of real-time data processing is its ability to facilitate fast
decision-making. As data enters the system, it is processed immediately and
provides on-the-spot insights. This capacity to analyze data on the fly allows you to
act and respond to situations as they unfold.

It also lets you handle customer queries or solve issues as they arise. This rapid
response mechanism, facilitated by real-time data processing, significantly enhances
customer service quality.

Reduction In Data Loss & Facilitated Data Recovery

Real-time data processing minimizes the risk of data loss as data is instantly saved
upon entry into the system. In the event of a system failure, real-time data
processing allows for immediate recovery of data because of its backup mechanism.

Rapid Processing Of Large Volumes Of Data

Real-time data processing supports scale-out processing, which ensures that it can
reliably handle large volumes of data. This becomes crucial in situations that require
the analysis or modification of large data sets within a tight timeframe.

Improved Customer Service & Trust Building

Real-time data processing helps you maintain updated user data and respond to
issues in real time which can dramatically enhance customer support. This way, you
can instantly provide customers with all the necessary data they need. It not only
builds trust but also takes customer service to a whole new level and you can forge
strong and meaningful relationships with your customers.

Swift Error Detection

When you use real-time data processing, you can detect errors and fix them quickly.
This not only helps prevent big or disastrous failures but also builds up the
company's reputation in the long run.

IV. Disadvantages Of Real-Time Data Processing (Đồng)

Financial Implications & Technical Demands

Real-time data processing requires specialized hardware and software which

significantly ramps up the overall complexity and cost. These systems are continually
operational and need regular investment and maintenance. This poses a financial
strain, especially for smaller organizations that can't afford the high cost of
acquisition and ongoing expenses.

Performance Limitations & Task Management

As data arrives in real-time, effective task prioritization becomes crucial to ensure

timely processing and avoid bottlenecks. One of the biggest challenges lies in
efficiently managing resources and allocating processing power to critical tasks,
particularly when multiple tasks compete for resources. As such, these systems are
less adaptable to task-switching, which makes it a challenge to manage and
complete tasks based on priority.

Security Risks & Accuracy Challenges

Real-time systems can pose security and privacy risks, especially when they are
handling sensitive or personal information. If not designed and implemented with
proper safeguards, there’s a risk of unauthorized access or data breaches.

When it comes to real-time processing, the need for speed and efficiency is a top
priority. However, there's a catch: the more we prioritize speed, the higher the risk of
errors. Even a small inaccuracy can have far-reaching consequences, especially in
critical sectors like financial trading or healthcare.

Limitations In Data Analysis

Real-time processing focuses primarily on handling current data which can limit its
ability to conduct historical analysis. The system may fail to identify long-term trends
and patterns that make it less ideal for strategic decision-making based on historical
data.

Additionally, it struggles with complex calculations, especially those involving data

averaging relative to other transactions. This lack of computational flexibility can be a
barrier in more complex data evaluation scenarios.

After weighing the pros and cons of real-time data processing, it's time to explore the
practical side of these theories and see how businesses are successfully using this
technology to optimize their operations and achieve extraordinary results.

V. Applications Of Real-Time Processing (Đồng)

Financial Trading
Real-time processing allows traders to make informed judgments based on current
data and market conditions. By digesting real-time data, traders may quickly
react to market developments and trade accordingly. This optimizes their
portfolio and profits.
Machine learning classifiers using high-quality real-time data can analyze real-
time market data and make predictions about stock prices. In fast-moving
markets, a few seconds’ delays can affect the outcome of a trade. Real-time
processing lets traders make more educated, strategic decisions, giving them a
market edge.

Fraud Detection
Real-time processing allows organizations to immediately identify and react to
suspicious activities. By looking at data in real-time, companies can set up
automated alerts for strange behavior or sudden increases in costs that they
didn’t expect.

For example, in the financial industry, an event hub can be used to monitor real-time
financial transactions for fraudulent activity. By analyzing the data in real time, the
event hub can send an alert to security personnel if it detects any unusual activity,
allowing them to quickly investigate and prevent financial loss.

Healthcare
Real-time processing can be used in healthcare to manage electronic health records
and data from medical systems. By processing and analyzing real-time data,
healthcare providers can quickly get information about a patient and make
treatment decisions based on that information.

A single-process algorithm could be used to continuously analyze the data from a

patient’s vital signs monitor and alert healthcare professionals if any values fall
outside of normal ranges. This allows for timely intervention and can potentially
prevent serious complications or even save a patient’s life.

Transportation
Real-time traffic control systems for processing transportation data enable the
monitoring and optimization of routes and the tracking of vehicles. With a huge
amount of GPS data available from tracking devices and cell phones, real-time
traffic updates can be generated and recommendations of alternative routes
can be made to drivers.

Real-time processing architecture can be used in radar systems and can enhance
public transportation schedules and routes and improve the customer experience. It
can also be used to update passengers on arrival times and delays, helping them
plan their journeys better.

Security And Surveillance

Real-time processing can greatly improve the ability of organizations to detect
threats, improving their security and surveillance systems. Real-time data along
with deep learning techniques can be used to analyze security camera video
feeds and report questionable activity.
Border and airport security use real-time processing to track people and threats. By
analyzing real-time data, these systems can respond quickly to possible threats,
ensuring human and facility safety.

VI. How Does Real-Time Processing Work? (Linh)

1. Data Generation:
- Data is generated continuously from various sources such as sensors, logs, user
interactions, or external systems.

2. Data Ingestion:
- The data is ingested into the real-time processing system. This step involves
collecting and receiving data from different sources. Common protocols for data
ingestion include MQTT, Kafka, and HTTP.

3. Data Processing:
- Once the data is ingested, it undergoes processing in real-time. This involves
transforming, enriching, and analyzing the data as it flows through the system.
- Streaming data processing frameworks, like Apache Flink, Apache Storm, or
Apache Kafka Streams, are often used for this purpose. These frameworks allow for
the parallel processing of data streams.

4. Storage (Optional):
- In some cases, the processed data might be stored in a real-time database or
data store for historical analysis or to serve other purposes.

5. Analytics and Decision Making:

- Real-time analytics tools and algorithms are applied to gain insights from the
processed data. This could involve monitoring for specific events, detecting patterns,
or running machine learning models to make predictions.
- Decision-making processes may also be integrated into the system, allowing
automated responses or triggering alerts based on predefined rules.

6. Output/Action:
- The insights gained from the real-time analysis can be used to trigger actions or
alerts. These actions could be automated responses, notifications to users, or
updates to other systems.

7. Feedback Loop:
- Real-time systems often have a feedback loop where the results of actions taken
are fed back into the system for further analysis and improvement.
VII. Real-Time Processing Tools (Linh)
The most prominent real-time processing tools:

1. Apache Kafka:
- Kafka is a distributed streaming platform that is commonly used for building real-
time data pipelines and streaming applications. It provides fault tolerance, scalability,
and durability of data.

2. Apache Flink:
- Flink is a stream processing framework that supports event time processing and
exactly-once semantics. It is designed for high-throughput, low-latency processing of
streaming data and supports stateful computations.

3. Apache Storm:
- Storm is a real-time stream processing system that is used for complex event
processing. It can be used for tasks such as real-time analytics, machine learning,
and continuous computation.

4. Amazon Kinesis:
- Amazon Kinesis is a cloud-based platform that enables real-time processing of
streaming data at scale. It includes services such as Kinesis Streams for real-time
data ingestion and Kinesis Analytics for real-time analytics.

5. Microsoft Azure Stream Analytics:

- Azure Stream Analytics is a fully managed, serverless real-time analytics service
provided by Microsoft Azure. It can process and analyze streaming data from IoT
devices, applications, and services.

6. Google Cloud Dataflow:

- Google Cloud Dataflow is a fully managed service for stream and batch
processing. It allows you to build both batch and stream processing pipelines using
Apache Beam, which is an open-source unified programming model for both.

Q&A (Chuẩn bị cho phần hỏi đáp)

● Phân biệt batch processing, stream processing, real-time processing
Real-time vs Batch Processing
Real-time data processing handles data as soon as possible, ingesting,
transforming, and exposing data products as soon as new data events are
generated. In contrast, batch processing handles data on some periodic
schedule. The differences between real-time data processing and batch data
processing should be apparent simply by name: One happens in “real-time”, and the
other happens in “batches”.

Real-Time Data Processing vs Stream Processing

Stream processing is a subset of real-time data processing that deals with limited
state and short time windows. Real-time data processing encompasses data
processing with large state over unbounded time windows. The core difference
between real-time data processing and stream processing is that real-time data
processing is optimized for large volumes of data stored over long periods.

● Nêu 1 ví dụ thực tiễn

●

Tài liệu tham khảo:

https://www.tinybird.co/blog-posts/real-time-data-processing (phân biệt batch vs
stream vs real-time processing)
https://hevodata.com/learn/real-time-processing/#stream
https://estuary.dev/what-is-real-time-processing/
https://insights.magestore.com/posts/du-lieu-lon-batch-processing-vs-streaming-
processing (tài liệu tiếng Việt)