BIG DATA ASSIGNMENT NOTES

ASSIGNMENT 3
1. How Does MapReduce Work in Hadoop?

MapReduce is a programming model used in Hadoop for processing large

data sets in a distributed manner.

How It Works:

🟩 Step 1: Input Splitting

 Large files are split into chunks (blocks).

 Each chunk is assigned to a Map task.

🟩 Step 2: Mapping

 Each Mapper processes a data block and produces key-value pairs.

 Example: Processing logs → (IP address, 1)

🟩 Step 3: Shuffling and Sorting

 Hadoop groups all values by key across all Mappers.

 Intermediate data is sorted and sent to Reducers.

🟩 Step 4: Reducing

 Reducers process each group of key-value pairs to produce final

output.

 Example: Summing counts per IP → (IP address, total visits)

🟩 Step 5: Output

 Final output is written to HDFS.

Example Use Case: Word count, log analysis, clickstream processing.

2. Difference Between HDFS and Traditional File Systems

 HDFS (Hadoop Distributed Traditional File System
Feature
File System) (e.g., NTFS, ext4)

Distributed across multiple

Architecture Centralized or single-machine
nodes

Fault Manual or external backup

Built-in data replication
Tolerance required

Scales horizontally (add more

Scalability Limited to hardware
nodes)

Data Size Optimized for large files (GBs

Not ideal for massive files
Handling to TBs)

Supports frequent read-write

Write Support Write-once, read-many
operations

Data Locality Computation moves to data Data moves to computation

Large (default 128 MB or 256 Smaller (4 KB – 64 KB

Block Size
MB) typically)

3. How Does Spark Compare to Hadoop for Big Data Processing?

Feature Apache Spark Hadoop (MapReduce)

Processing In-memory Disk-based

Up to 100x faster for some Slower due to frequent disk

Speed
workloads I/O

High-level APIs (Python, Requires Java-based

Ease of Use
Scala, Java, R) MapReduce code

Real-time
Yes (Spark Streaming) No (batch only)
Support

Machine Limited support (needs

Built-in MLlib
Learning external tools)

Through task re-execution

Fault Tolerance DAG lineage and RDDs
and replication

Data Processing Batch, Streaming, Only batch processing

Feature Apache Spark Hadoop (MapReduce)

Modes Interactive, Graph

Summary:

 Use Hadoop MapReduce for batch jobs on extremely large datasets.

 Use Apache Spark for faster, in-memory, interactive or real-time

data processing.

UNIT 4
1. What is NoSQL, and How is it Used in Big Data Storage?

✅ Definition:

NoSQL (Not Only SQL) databases are non-relational databases designed

to handle large volumes of unstructured, semi-structured, or
structured data with high performance and scalability.

✅ Types of NoSQL Databases:

Type Description Examples

Document- Stores data as JSON-like

MongoDB, CouchDB
based documents

Redis, Amazon
Key-Value Stores pairs for fast lookups
DynamoDB

Stores data in columns instead Apache Cassandra,

Column-based
of rows HBase

Optimized for Neo4j, Amazon

Graph-based
relationships/networks Neptune

✅ Use in Big Data:

 Handles high volume, velocity, and variety of data.

 Scales horizontally across distributed clusters.

 Useful in real-time analytics, IoT, recommendation systems, and social

media platforms.
2. How Do You Handle Data Quality Issues in Big Data Sets?

Big data often contains noise, duplication, or missing values. Here's how you
can manage quality issues:

✅ Steps to Handle Data Quality:

Issue Type Handling Techniques

Imputation (mean/median), data interpolation,

Missing Data
deletion

Duplicate
Use hashing or unique IDs to remove duplicates
Records

Inconsistent Standardize units (e.g., date formats, case

Formats normalization)

Use statistical or ML techniques to detect and

Outliers/Noise
handle

Incorrect Data Cross-validation with reference datasets or rules

✅ Tools Commonly Used:

 Apache Spark, Talend, Trifacta, OpenRefine, Pandas (in Python)

3. Techniques for Data Preprocessing in Big Data

Data preprocessing prepares raw data for analytics or machine learning

models.

✅ Common Techniques:

Technique Purpose

Fix/remove incorrect, incomplete, or inconsistent

Data Cleaning
data

Data
Normalize, scale, encode data for algorithms
Transformation

Combine data from multiple sources (ETL

Data Integration
processes)

Data Reduction Dimensionality reduction (e.g., PCA), sampling,

Technique Purpose

aggregation

Convert continuous data into categories or

Data Discretization
intervals

Tokenization &
For text data — splitting sentences into words
Parsing

Streaming Real-time data transformation using tools like

Preprocessing Kafka, Spark

✅ Big Data Tools for Preprocessing:

 Apache Spark (with PySpark or Scala)

 Apache NiFi

 Hadoop MapReduce

 ETL pipelines (Airflow, Talend)

UNIT 5
✅ 1. How Do You Implement Data Governance in a Big Data
Environment?

Data governance ensures that data is accurate, secure, consistent, and

used responsibly.

📌 Key Components of Data Governance in Big Data:

Component Description

Data Catalog Centralized metadata store (e.g., Apache Atlas, Alation)

Tracks data flow from source to destination (e.g.,

Data Lineage
OpenLineage, Talend)

Role-Based Access Control (RBAC), policies for who can

Access Control
access what

Data Quality
Define valid values, types, ranges, null handling
Rules

Data Assign responsible roles for maintaining data integrity

Component Description

Stewardship

Policy
Compliance with GDPR, HIPAA, etc.
Management

🔧 Tools for Data Governance:

 Apache Atlas (metadata management)

 Apache Ranger (fine-grained access control)

 Collibra, Informatica, AWS Glue Data Catalog

✅ 2. Common Big Data Security Threats & Mitigation Strategies

⚠️Common Security Threats:

Threat Description Mitigation Strategies

Unauthorized access to Encryption (at-rest/in-transit),

Data Breaches
sensitive data Access controls

Unauthorized Lack of strict access Use Kerberos, LDAP, or OAuth

Access policies authentication

Data Leakage in Leakage during Secure APIs, TLS/SSL, audit

Pipelines processing or transfers trails

Malicious Code Attacks via open-source Code scanning, sandboxing

Injection or shared scripts jobs

Lack of Audit No monitoring of data Use logging systems like

Trails usage Apache Ranger, audit tools

🔐 Key Security Techniques:

 Kerberos: Secure authentication in Hadoop/Spark

 Apache Ranger: Role-based policies and audit logs

 Tokenization & Encryption: Protects PII data

 Network Layer Security: VPN, firewalls, VPCs

✅ 3. How Do You Scale Big Data Processing for Real-Time Analytics?

Real-time analytics requires fast ingestion, low-latency processing, and

scalable architecture.

⚙️Architecture for Real-Time Analytics:

text

CopyEdit

[Data Sources]

[Ingestion Layer] — Kafka / Flume / Kinesis

[Processing Layer] — Apache Spark Streaming / Flink / Storm

[Storage Layer] — Cassandra / HBase / Elasticsearch

[Visualization Layer] — Grafana / Kibana / Tableau

🧠 Key Techniques:

Technique Purpose

Real-time data computation (Spark

Stream Processing
Streaming, Flink)

Micro-Batching Efficient processing in small time windows

Autoscaling Dynamic resource allocation in cloud (K8s,

Infrastructure EMR)

Event-Driven
Process events instantly via Kafka or Pulsar
Architecture

In-Memory
Fast processing using RAM (Spark, Ignite)
Computing

🛠 Example Tools:

 Kafka + Spark Structured Streaming for low-latency pipelines

  AWS Kinesis + Lambda for serverless real-time processing

 Apache Flink for advanced stream processing with stateful operators

🚀 What is Apache Spark?

Apache Spark is an open-source, distributed computing framework

designed for fast processing of large-scale data. It supports batch
processing, streaming, machine learning, and SQL-based analytics —
all in one platform.

Feature Description

In-Memory Keeps intermediate data in memory for faster processing

Computing than Hadoop MapReduce

Supports SQL, MLlib (Machine Learning), GraphX (Graph

Unified Engine
Processing), and Spark Streaming

Language
APIs available in Python, Scala, Java, and R
Support

Distributed
Splits tasks across a cluster for parallel execution
Computing

Fault Tolerant Automatically handles failures using RDD lineage

🔄 Spark vs Hadoop (MapReduce)

Feature Apache Spark Hadoop MapReduce

Speed Faster (in-memory) Slower (disk-based)

Rich APIs in
Ease of Use Low-level Java APIs
Python/Scala/Java

Data
Batch + Streaming Batch only
Processing

Machine Needs integration with external

Built-in MLlib
Learning tools
🔥 Core Components of Apache Spark

1. Spark Core – The execution engine (RDDs, memory mgmt, fault

tolerance)

2. Spark SQL – Query structured data using SQL or DataFrames

3. Spark Streaming – Real-time data processing from sources like Kafka

4. MLlib – Machine learning library (classification, clustering, etc.)

5. GraphX – Graph processing (e.g., PageRank, graph traversal)

💡 Example: PySpark Code for Word Count

python

CopyEdit

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("WordCount").getOrCreate()

# Load text file

rdd = spark.sparkContext.textFile("sample.txt")

# Word count logic

counts = (

rdd.flatMap(lambda line: line.split())

.map(lambda word: (word, 1))

.reduceByKey(lambda a, b: a + b)

counts.collect()
✅ Common Use Cases

 Real-time analytics (e.g., fraud detection, log monitoring)

 ETL (Extract, Transform, Load) pipelines

 Recommendation engines

 Social media data analysis

 IoT stream processing

Would you like:

 A simple Spark ML example (like linear regression)?

 A real-time example using Spark Streaming + Kafka?

🚀 Apache Spark Architecture: Overview

Apache Spark follows a master-slave architecture with the following key

components:

🧱 Core Components of Apache Spark:

Component Role

Driver Controls the application, manages SparkContext, and

Program coordinates tasks

Cluster Allocates resources across Spark applications (e.g., YARN,

Manager Mesos, Kubernetes)

Executors Run tasks and return results to the driver

Tasks Individual units of work sent to executors

🔧 Detailed Components of Apache Spark

1. Spark Core (Foundation of everything)

 Manages memory, fault-tolerance, job scheduling.

 Provides the RDD (Resilient Distributed Dataset) abstraction for

distributed data.
2. Spark SQL

 Allows querying structured and semi-structured data using SQL,

DataFrames, and Datasets.

 Can read from Hive, Parquet, JSON, JDBC, etc.

3. Spark Streaming

 Enables real-time data processing.

 Processes live data streams using micro-batching.

4. MLlib (Machine Learning Library)

 Built-in library for scalable machine learning tasks:

o Classification, Regression, Clustering, Recommendation

5. GraphX

 API for graph processing (e.g., social networks, recommendation

graphs).

 Includes graph algorithms like PageRank and connected components.

⚙️How Apache Spark Works (Step-by-Step Execution)

Let’s understand with an example:

💼 Suppose: You want to count words in a large text file using Spark.

🔁 Spark Job Workflow:

1. Driver Program Starts

o It creates a SparkContext (entry point to Spark cluster).

2. Cluster Manager Allocates Resources

o The driver asks for executors on cluster nodes.

3. RDD/DataFrame Created

o Data is loaded into an RDD (e.g., from a text file).

4. Transformations Applied

o Operations like .map(), .filter(), .flatMap() define a DAG (Directed

Acyclic Graph).
 5. Actions Trigger Execution

o An action like .collect() or .saveAsTextFile() starts actual

processing.

6. Task Scheduling

o Spark breaks the DAG into stages and tasks.

7. Tasks Sent to Executors

o Executors perform computations in parallel.

8. Results Returned

o Executors return the results to the driver, or write to storage.

🖼 Spark Architecture Diagram (Text-Based)

plaintext

CopyEdit

+----------------------+

| Driver Program | ← Controls the job

+----------------------+

+----------------------+ +----------------------+

| Cluster Manager | ←→→→ | Executors (n) |

+----------------------+ +----------------------+

| |

v v

Distribute Tasks Process Data, Store Cache

🧠 Summary
Component Responsibility

Main controller, builds job, sends tasks to

Driver
workers

Executor Workers that run tasks and store data

Cluster
Manages resources and task scheduling
Manager

RDD/
Data abstraction used for processing
DataFrame