Top 70+ Data Engineer Interview Questions

and Answers​
1. What is Data Engineering?​
This may seem like a pretty basic data engineer interview questions, but regardless of your skill
level, this may come up during your interview. Your interviewer wants to see what your specific
definition of data engineering is, which also makes it clear that you know what the work entails.
So, what is it? In a nutshell, it is the act of transforming, cleansing, profiling, and aggregating
large data sets. You can also take it a step further and discuss the daily duties of a data engineer,
such as ad-hoc data query building and extracting, owning an organization’s data stewardship,
and so on.​
2. Why did you choose a career in Data Engineering?​
An interviewer might ask this question to learn more about your motivation and interest behind
choosing data engineering as a career. They want to employ individuals who are passionate
about the field. You can start by sharing your story and insights you have gained to highlight
what excites you most about being a data engineer. ​
3. How does a data warehouse differ from an operational database?​
This data engineer interview question may be more geared toward those on the intermediate
level, but in some positions, it may also be considered an entry-level question. You’ll want to
answer by stating that databases using Delete SQL statements, Insert, and Update is standard
operational databases that focus on speed and efficiency. As a result, analyzing data can be a
little more complicated. With a data warehouse, on the other hand, aggregations, calculations,
and select statements are the primary focus. These make data warehouses an ideal choice for
data analysis.​
4. As a data engineer, how have you handled a job-related crisis?​
Data engineers have a lot of responsibilities, and it’s a genuine possibility that you’ll face
challenges while on the job, or even emergencies. Just be honest and let them know what you
did to solve the problem. If you have yet to encounter an urgent issue while on the job or this is
your first data engineering role, tell your interviewer what you would do in a hypothetical
situation. For example, you can say that if data were to get lost or corrupted, you would work
with IT to make sure data backups were ready to be loaded, and that other team members have
access to what they need.​
5. Do you have any experience with data modeling?​
Unless you are interviewing for an entry-level role, you will likely be asked this question at some
point during your interview. Start with a simple yes or no. Even if you don’t have experience
with data modeling, you’ll want to be at least able to define it: the act of transforming and
processing fetched data and then sending it to the right individual(s). If you are experienced, you
can go into detail about what you’ve done specifically. Perhaps you used tools like Talend,
Pentaho, or Informatica. If so, say it. If not, simply being aware of the relevant industry tools and
what they do would be helpful.​
6. Why are you interested in this job, and why should we hire you? ​
It is a fundamental data engineer interview question, but your answer can set you apart from
the rest. To demonstrate your interest in the job, identify a few exciting features of the job, which
makes it an excellent fit for you and then mention why you love the company. ​
For the second part of the question, link your skills, education, personality, and professional
experience to the job and company culture. You can back your answers with examples from
previous experience. As you justify your compatibility with the job and company, be sure to
depict yourself as energetic, confident, motivated, and culturally fit for the company. ​
7. What are the essential skills required to be a data engineer?​
Every company can have its own definition of a data engineer, and they match your skills and
qualifications with the company's assessment. ​
Here is a list of must-have skills and requirements if you are aiming to be a successful data
engineer:​
• Comprehensive knowledge about Data Modelling.​
• Understanding about database design & database architecture. In-Depth Database
Knowledge – SQL and NoSQL.​
• Working experience of data stores and distributed systems like Hadoop (HDFS).​
• Data Visualization Skills.​
• Experience in Data Warehousing and ETL (Extract Transform Load) Tools.​
• You should have robust computing and math skills.​
• Outstanding communication, leadership, critical thinking, and problem-solving capabilities
are an added advantage. ​
You can mention specific examples in which a data engineer would apply these skills.​
8. Can you name the essential frameworks and applications for
data engineers?​
This data engineer interview question is often asked to evaluate whether you understand the
critical requirements for the position and have the desired technical skills. In your answer,
accurately mention the names of frameworks along with your level of experience with each. ​
You can list all of the technical applications like SQL, Hadoop, Python, and more, along with
your proficiency level in each. You can also state the frameworks which want to learn more
about if given the opportunity.​
9. Are you experienced in Python, Java, Bash, or other scripting
languages?​
This question is asked to emphasize the importance of understanding scripting languages as a
data engineer. It is essential to have a comprehensive knowledge of scripting languages, as it
allows you to perform analytical tasks efficiently and automate data flow.​
10. Can you differentiate between a Data Engineer and Data
Scientist?​
With this question, the recruiter is trying to assess your understanding of different job roles
within a data warehouse team. The skills and responsibilities of both positions often overlap,
but they are distinct from each other. ​
Data Engineers develop, test, and maintain the complete architecture for data generation,
whereas data scientists analyze and interpret complex data. They tend to focus on organization
and translation of Big Data. Data scientists require data engineers to create the infrastructure for
them to work.​
11. What, according to you, are the daily responsibilities of a data
engineer?​
This question assesses your understanding of the role of a data engineer role and job
description. ​
You can explain some crucial tasks a data engineer like:​
• Development, testing, and maintenance of architectures.​
• Aligning the design with business requisites.​
• Data acquisition and development of data set processes.​
• Deploying machine learning and statistical models​
• Developing pipelines for various ETL operations and data transformation​
• Simplifying data cleansing and improving the de-duplication and building of data.​
• Identifying ways to improve data reliability, flexibility, accuracy, and quality.​
This is one of the most commonly asked data engineer interview questions.​
12. What is your approach to developing a new analytical product
as a data engineer?​
The hiring managers want to know your role as a data engineer in developing a new product and
evaluate your understanding of the product development cycle. As a data engineer, you control
the outcome of the final product as you are responsible for building algorithms or metrics with
the correct data. ​
Your first step would be to understand the outline of the entire product to comprehend the
complete requirements and scope. Your second step would be looking into the details and
reasons for each metric. Think about as many issues that could occur, and it helps you to create
a more robust system with a suitable level of granularity.​
13. What was the algorithm you used on a recent project?​
The interviewer might ask you to select an algorithm you have used in the past project and can
ask some follow-up questions like:​
• Why did you choose this algorithm, and can you contrast this with other similar ones?
• What is the scalability of this algorithm with more data? ​
• Are you happy with the results? If you were given more time, what could you improve?​
These questions are a reflection of your thought process and technical knowledge. First, identify
the project you might want to discuss. If you have an actual example within your area of
expertise and an algorithm related to the company's work, then use it to pique the interest of
your hiring manager. Secondly, make a list of all the models you worked with and your analysis.
Start with simple models and do not overcomplicate things. The hiring managers want you to
explain the results and their impact.​
14. What tools did you use in a recent project?​
Interviewers want to assess your decision-making skills and knowledge about different tools.
Therefore, use this question to explain your rationale for choosing specific tools over others. ​
• Walk the hiring managers through your thought process, explaining your reasons for
considering the particular tool, its benefits, and the drawbacks of other technologies. ​
• If you find that the company works on the techniques you have previously worked on, then
weave your experience with the similarities.​
15. What challenges came up during your recent project, and how
did you overcome these challenges?​
Any employer wants to evaluate how you react during difficulties and what you do to address
and successfully handle the challenges. ​
When you talk about the problems you encountered, frame your answer using the STAR method:​
• Situation: Brief them about the circumstances due to which problem occurred.​
• Task: It is essential to elaborate on your role in overcoming the problem. For example, if you
took a leadership role and provided a working solution, then showcasing it could be decisive
if you were interviewing for a leadership position.​
• Action: Walk the interviewer through the steps you took to fix the problem. ​
• Result: Always explain the consequences of your actions. Talk about the learnings and
insights gained by you and other stakeholders.​
16. Have you ever transformed unstructured data into structured
data?​
It is an important question as your answer can demonstrate your understating of both the data
types and your practical working experience. You can answer this question by briefly
distinguishing between both categories. The unstructured data must be transformed into
structured data for proper data analysis, and you can discuss the methods for transformation.
You must share a real-world situation wherein you changed the unstructured data into
structured data. If you are a fresh graduate and don't have professional experience, discuss
information related to your academic projects.​
17. What is Data Modelling? Do you understand different Data
Models?​
Data Modelling is the initial step towards data analysis and database design phase. Interviewers
want to understand your knowledge. You can explain that is the diagrammatic representation to
show the relation between entities. First, the conceptual model is created, followed by the
logical model and, finally, the physical model. The level of complexity also increases in this
pattern. ​
18. Can you list and explain the design schemas in Data Modelling?​
Design schemas are the fundamentals of data engineering, and interviewers ask this question to
test your data engineering knowledge. In your answer, try to be concise and accurate. Describe
the two schemas, which are Star schema and Snowflake schema. ​
Explain that Star Schema is divided into a fact table referenced by multiple dimension tables,
which are all linked to a fact table. In contrast, in Snowflake Schema, the fact table remains the
same, and dimension tables are normalized into many layers looking like a snowflake.​
19. How would you validate a data migration from one database to
another?​
The validity of data and ensuring that no data is dropped should be of utmost priority for a data
engineer. Hiring managers ask this question to understand your thought process on how
validation of data would happen. ​
You should be able to speak about appropriate validation types in different scenarios. For
instance, you could suggest that validation could be a simple comparison, or it can happen after
the complete data migration. ​
20. Have you worked with ETL? If yes, please state, which one do
you prefer the most and why?​
With this question, the recruiter needs to know your understanding and experience regarding
the ETL (Extract Transform Load) tools and process. You should list all the tools in which you
have expertise and pick one as your favourite. Point out the vital properties which make that
tool stand out and validate your preference to demonstrate your knowledge in the ETL process.​
21. What is Hadoop? How is it related to Big data? Can you describe
its different components?​
This question is most commonly asked by hiring managers to verify your knowledge and
experience in data engineering. You should tell them that Big data and Hadoop are related to
each other as Hadoop is the most common tool for processing Big data, and you should be
familiar with the framework. ​
With the escalation of big data, Hadoop has also become popular. It is an open-source software
framework that utilizes various components to process big data. The developer of Hadoop is the
Apache foundation, and its utilities increase the efficiency of many data applications. ​
Hadoop comprises of mainly four components: ​
1. HDFS stands for Hadoop Distributed File System and stores all of the data of Hadoop. Being a
distributed file system, it has a high bandwidth and preserves the quality of data.
2. MapReduce processes large volumes of data.​
3. Hadoop Common is a group of libraries and functions you can utilize in Hadoop.​
4. YARN (Yet Another Resource Negotiator)deals with the allocation and management of
resources in Hadoop. ​
5. Do you have any experience in building data systems using the Hadoop framework? ​
If you have experience with Hadoop, state your answer with a detailed explanation of the work
you did to focus on your skills and tool's expertise. You can explain all the essential features of
Hadoop. For example, you can tell them you utilized the Hadoop framework because of its
scalability and ability to increase the data processing speed while preserving the quality.​
Some features of Hadoop include: ​
• It is Java-Based. Hence, there may be no additional training required for team members.
Also, it is easy to use.
• As the data is stored within Hadoop, it is accessible in the case of hardware failure from other
paths, which makes it the best choice for handling big data. ​
• In Hadoop, data is stored in a cluster, making it independent of all the other operations.​
In case you have no experience with this tool, learn the necessary information about the tool's
properties and attributes.​
22. Can you tell me about NameNode? What happens if NameNode
crashes or comes to an end?​
It is the centre-piece or central node of the Hadoop Distributed File System(HDFS), and it does
not store actual data. It stores metadata. For example, the data being stored in DataNodes on
which rack and which DataNode the information is stored. It tracks the different files present in
clusters. Generally, there is one NameNode, so when it crashes, the system may not be available.​
23. Are you familiar with the concepts of Block and Block Scanner in
HDFS?​
You'll want to answer by describing that Blocks are the smallest unit of a data file. Hadoop
automatically divides huge data files into blocks for secure storage. Block Scanner validates the
list of blocks presented on a DataNode.​
24. What happens when Block Scanner detects a corrupted data
block?​
It is one of the most typical and popular interview questions for data engineers. You should
answer this by stating all steps followed by a Block scanner when it finds a corrupted block of
data. ​
Firstly, DataNode reports the corrupted block to NameNode.NameNode makes a replica using
an existing model. If the system does not delete the corrupted data block, NameNode creates
replicas as per the replication factor. ​
25. What are the two messages that NameNode gets from
DataNode?​
NameNodes gets information about the data from DataNodes in the form of messages or
signals. ​
The two signs are:​
1. Block report signals which are the list of data blocks stored on DataNode and its functioning.​
2. Heartbeat signals that the DataNode is alive and functional. It is a periodic report to establish
whether to use NameNode or not. If this signal is not sent, it implies DataNode has stopped
working.​
26. Can you elaborate on Reducer in Hadoop MapReduce? Explain
the core methods of Reducer?​
Reducer is the second stage of data processing in the Hadoop Framework. The Reducer
processes the data output of the mapper and produces a final output that is stored in HDFS. ​
The Reducer has 3 phases:​
1. Shuffle: The output from the mappers is shuffled and acts as the input for Reducer.​
2. Sorting is done simultaneously with shuffling, and the output from different mappers is
sorted.
3. Reduce: in this step, Reduces aggregates the key-value pair and gives the required output,
which is stored on HDFS and is not further sorted.​
There are three core methods in Reducer:​
1. Setup: it configures various parameters like input data size.​
2. Reduce: It is the main operation of Reducer. In this method, a task is defined for the
associated key.​
3. Cleanup: This method cleans temporary files at the end of the task.​
4. How can you deploy a big data solution?​
While asking this question, the recruiter is interested in knowing the steps you would follow to
deploy a big data solution. You should answer by emphasizing on the three significant steps
which are:​
1. Data Integration/Ingestion: In this step, the extraction of data using data sources like RDBMS,
Salesforce, SAP, MySQL is done.​
2. Data storage: The extracted data would be stored in an HDFS or NoSQL database.​
3. Data processing: the last step should be deploying the solution using processing frameworks
like MapReduce, Pig, and Spark.​
27. Which Python libraries would you utilize for proficient data
processing?​
This question lets the hiring manager evaluate whether the candidate knows the basics of
Python as it is the most popular language used by data engineers. ​
Your answer should include NumPy as it is utilized for efficient processing of arrays of numbers
and pandas, which is great for statistics and data preparation for machine learning work. The
interviewer can ask you questions like why would you use these libraries and list some examples
where you would not use them.​

28. Can you differentiate between list and tuples?​

Again, this question assesses your in-depth knowledge of Python. In Python, List and Tuple are
the classes of data structure where Lists are mutable and can be edited, but Tuples are
immutable and cannot be modified. Support your points with the help of examples.​
29. How can you deal with duplicate data points in an SQL query?​
Interviewers can ask this question to test your SQL knowledge and how invested you are in this
interview process as they would expect you to ask questions in return. You can ask them what
kind of data they are working with and what values would likely be duplicated? ​
You can suggest the use of SQL keywords DISTINCT & UNIQUE to reduce duplicate data points.
You should also state other ways like using GROUP BY to deal with duplicate data points.​
30. Did you ever work with big data in a cloud computing
environment?​
Nowadays, most companies are moving their services to the cloud. Therefore, hiring managers
would like to understand your cloud computing capabilities, knowledge of industry trends, and
the future of the company's data. ​
You must answer it stating that you are prepared for the possibility of working in a virtual
workspace as it offers many advantages like:​
• Flexibility to scale up the environment as required, ​
• Secure access to data from anywhere​
• Having backups in case of an emergency
31. How can data analytics help the business grow and boost
revenue?​
Ultimately, it all comes down to business growth and revenue generation, and Big Data analysis
has become crucial for businesses. All companies want to hire candidates who understand how
to help the business grow, achieve their goals, and result in higher ROI. ​
You can answer this question by illustrating the advantages of data analytics to boost revenue,
improve customer satisfaction, and increase profit. Data analytics helps in setting realistic goals
and supports decision making. By implementing Big Data analytics, businesses may encounter a
5-20% significant increase in revenue. Walmart, Facebook, LinkedIn are some of the companies
using big data analytics to boost their income.​
32. Define Hadoop Streaming.
Hadoop Streaming is a feature or utility included with a Hadoop distribution that lets
programmers or developers construct Map-Reduce programs in many programming languages
such as Python, Ruby, C++, Perl, and others. We may leverage any language capable of reading
from STDIN (standard input), such as keyboard input, and write to STDOUT (standard output).​
33. What is the full form of HDFS?​
The full form of HDFS is Hadoop Distributed File System.​
34. List out various XML configuration files in Hadoop.​
The following are the various XML configuration files in Hadoop:​
• HADOOP-ENV.sh​
• CORE-SITE.XML​
• HDFS-SITE.XML​
• MAPRED-SITE.XML​
• Masters​
• Slave​

35. What are the four v’s of big data?​

The four V's of Big Data are Volume, Velocity, Variety, and Veracity.​
36. Explain the features of Hadoop.​
Some of the most important features of Hadoop are:​
• It's open-source: It is an open-source project, which implies that its source code is freely
available for modification, inspection, and analysis, allowing organisations to adapt the code
to meet their needs.​
• It offers fault tolerance: Hadoop's most critical feature is fault tolerance. To achieve fault
tolerance, Hadoop 2's HDFS employs a replication strategy. Based on the replication, it
beautifully makes a clone of every block on each system (by default, it’s 3). As a result, if
any machine in a cluster fails, data may be accessed from other devices that carry duplicates
of the same data.​
• It is highly scalable: To reach high computing power, the Hadoop cluster is highly scalable,
which means we may add any amount of nodes or expand the hardware potential of nodes.
This gives the Hadoop architecture horizontal as well as vertical scalability.​
37. What is the abbreviation of COSHH?​
COSHH stands for Control of Substances Hazardous to Health.​
38. Explain Star Schema.​
A Star Schema is basically a multi-dimensional data model that is used to arrange data in a
database so that it may be easily understood and analysed. Data marts, Data Warehouses,
databases, and other technologies can all benefit from star schemas. The star schema style is
ideal for querying massive amounts of data.​
39. Explain FSCK​
FSCK, an acronym for File System Consistency Checker, is one method older Linux-based
systems still employ to detect and correct errors. It is not a comprehensive solution, as inodes
pointing to junk data may still exist. The primary goal is to ensure that the metadata is internally
consistent.​
40. Explain Snowflake Schema.​
A snowflake schema is basically a multidimensional database schema that divides
subdimensions into dimension tables. Engineers convert every dimension table into logical
subdimensions while designing a snowflake schema. As a result, the data model turns out to be
more complicated, but it might also make it straightforward for analysts in dealing with it,
particularly for certain data kinds. Because its ERD (entity-relationship diagram) resembles a
snowflake, it is known as the "snowflake schema."​
41. Distinguish between Star and Snowflake Schema.​
The following are some of the distinguishing features of a StE Schema and a Snowflake Schema:​
• The star schema is the most basic kind of Data Warehouse schema. It's referred to as the star
schema as its structure is similar to a star's. A Snowflake Schema is an expansion of a Star
Schema that adds dimension. It's called a snowflake, as the diagram appears to be like a
snowflake.​
• Only a single join describes the link between any dimension table and the fact table in a star
schema. A fact table is enveloped by dimension tables in the star schema, whereas a
snowflake schema is surrounded by dimension tables, which are surrounded by dimension
tables, and so forth. To get data from a snowflake schema, numerous joins are required.​
42. Explain Hadoop Distributed File System.​
Hadoop applications use HDFS (Hadoop Distributed File System) as their primary storage
system. This open-source framework operates by passing data between nodes as quickly as
possible. Companies that must process and store large amounts of data frequently employ it.
HDFS is a critical component of many Hadoop systems since it allows for managing and
analysing large amounts of data.​

43. What Is the full form of YARN?​

The full form of YARN is Yet Another Resource Negotiator.​
44. List various modes in Hadoop.​
There are three different types of modes in Hadoop:​
• Fully-Distributed Mode​
• Pseudo-Distributed Mode​
• Standalone Mode​

45. How to achieve security in Hadoop?​

Apache Hadoop offers users security in the following ways:​
• Kerberos was implemented using SASL/GSSAPI. It is also used on RPC connections to
mutually validate users, their procedures, and Hadoop services.​
• Delegation tokens are used in connection with the NameNode for future authenticated
access that does not need the usage of the Kerberos Server.​
• Web application and web console developers might create their own HTTP authentication
method, including HTTP SPNEGO authentication.​
46. What Is Heartbeat in Hadoop?​
A heartbeat in Hadoop is a signal sent from the Datanode to the Namenode, indicating it is alive.
In HDFS, the lack of a heartbeat signals a problem, and the Namenode and Datanode cannot do
any computations.​
47. Distinguish between NAS and DAS in Hadoop.​
The following are some of the differences between NAS (Network Attached Storage) and DAS
(Direct Attached Storage):​
• The computing and storage layers are separated in NAS. Storage is dispersed among several
servers in a network. Storage is tied to the node where computing occurs in DAS.​
• Apache Hadoop is founded on the notion of bringing processing close to the data. As a result,
the storage disc must be close to the calculation. DAS provides excellent performance on a
Hadoop cluster. DAS can also be implemented on common hardware. As a result, it is less
expensive when compared to NAS.​
48. List important fields or languages used by data engineers.​
Scala, Java, and Python are some of the most sought-after programming languages that are
leveraged by data engineers.​
49. What is Big Data?​
Big data refers to huge, complicated data sets that are created and sent in real-time from a wide
range of sources. Big data collections can be organised, semi-structured, or unstructured, and
they are regularly examined to uncover relevant patterns and insights regarding user and
machine behaviour.​
50. What is FIFO Scheduling?​
FCFS, or First Come First Service, is basically an operating system scheduling method that
performs queued processes and requests in the order in which they arrive. It's the most
straightforward and basic CPU scheduling technique. Processes seeking the CPU first receive the
CPU allocation in this method. A FIFO queue is leveraged to handle this.​
Mention default port numbers on which the task tracker, NameNode, and job tracker run in
Hadoop.​
• Task Tracker: 50060​
• NameNode: 50070​
• JobTracker: 50030​

51. How to define the distance between two nodes in Hadoop?​

The network is represented as a tree in Hadoop. The distance between two nodes is the total of
their ancestor distances.​
52. Why use commodity hardware in Hadoop?​
The idea behind using Commodity Hardware in Hadoop is simple: you've got a few servers and
distribute the load among them. It is possible because of Hadoop MapReduce, a fantastic
component of this setup.​
In Hadoop, commodity hardware is put on all servers, and it then distributes data across them.
Every server will eventually hold a piece of data. No server, however, will contain everything.​
53. Define Replication Factor in HDFS.​
The replication factor specifies the number of copies of a block that should be stored in your
cluster. Because the replication factor is set to three by default, every file you create in the
Hadoop Distributed File System will be having a replication factor of three, and each block in the
file will be duplicated to three distinct nodes in your cluster.​
54. What data is stored in NameNode?​
NameNode serves as the master of the system. It monitors the metadata and file system tree for
every folder and file on the system. The information of metadata is saved in two files: 'Edit Log'
and 'Namespace image'.​
55. What do you mean by Rack Awareness?​
Rack awareness in Hadoop refers to recognising how various data nodes are dispersed across
racks or knowing the cluster architecture in the Hadoop cluster.​
56. What are the functions of Secondary NameNode?​
The following are some of the functions of the secondary NameNode:​
• Keeps a copy of the FsImage file and an edit log.​
• Apply edits log entries to the FsImage file on a regular basis and renews the edits log. It then
sends this modified FsImage file directly to NameNode so that it does not have to re-address
the EditLog records at the time of the startup process. As a result, Secondary NameNode
speeds up the NameNode startup procedure.​
• If NameNode fails, File System information can be retrieved from the last stored FsImage on
the Secondary NameNode, however, the Secondary NameNode cannot take over the
functionality of the Primary NameNode.​
• File system information is checked for accuracy.​
57. What are the basic phases of reducer in Hadoop?​
A Reducer in Hadoop has three major phases:​
• Shuffle: Reducer duplicates the sorted output from each Mapper during this step.​
• Sort: During this stage, the Hadoop framework sorts the input to the Reducer by the same
key. This step employs merge sort. Sometimes the shuffle and sort processes occur
concurrently.​
• Reduce: It's the stage at which the output values associated with a key are lowered to
produce an output result. Reducer output is not re-sorted.​
58. Why does Hadoop use Context objects?​
Context object permits the Mapper/Reducer to communicate with the remainder of the Hadoop
system. It contains task configuration data and interfaces that allow it to send output. It can be
used by programs to report progress.​
59. Define Combiner in Hadoop.​
The Combiner, also known as the "Mini-Reducer," summarises the Mapper output record using
the same Key before handing it to the Reducer.​
When we execute a MapReduce task on a huge dataset. As a result, Mapper creates vast amounts
of intermediate data. The framework then forwards this intermediate data to the Reducer for
further processing.​
This causes massive network congestion. The Hadoop framework has a function called
Combiner, which helps to reduce network congestion.​
Combiner, sometimes known as a "Mini-Reducer," is responsible for processing the output data
from the Mapper before transferring it to the Reducer. It is executed after the mapper but before
the reducer. Its application is discretionary.​
60. What is the default replication factor available in HDFS? What
does it indicate?​
HDFS's replication factor is set to 3 by default. This implies that each block will have two
additional copies stored on a different DataNode in the cluster.​
61. What do you mean by Data Locality in Hadoop?​
Data locality in Hadoop brings computation near where the real data is on the node rather than
transporting massive data to computation. It lowers network congestion while increasing total
system throughput.​
62. Define Balancer in HDFS.​
The HDFS Balancer is basically a utility for balancing data across an HDFS cluster's storage
devices. The HDFS Balancer was initially designed to run slowly so that balancing operations did
not interfere with regular cluster activity and job execution.​
63. Explain Safe Mode in HDFS.​
The Hadoop Distributed File System (HDFS) cluster's safe mode for the NameNode is read-only.
You cannot change or block the file system while in Safe Mode. When the DataNodes indicate
that most file system blocks are accessible, the NameNode exits Safe Mode automatically.​
64. What is the importance of Distributed Cache in Apache Hadoop?​
In Hadoop, the distributed cache is a method of copying archives or small files to worker nodes
in real time. This is done by Hadoop so that these worker nodes may use them when conducting
a job. To conserve network traffic, files are copied just once per job.​
65. What is Metastore in Hive?​
Metastore in Hive is the component that maintains all of the warehouse's structural information,
including serializers and deserializers required to read and write data, column and column type
information, and the accompanying HDFS files where the data is kept.​
66. What do you mean by SerDe in Hive?​
Athena communicates with data in multiple forms via a SerDe (Serializer/Deserializer). The
SerDe you provide defines the table schema, not the DDL. In other words, the SerDe can overrule
the DDL settings you give when you create your table in Athena.​
67. List the components available in the Hive data model.​
The following components are included in Hive data models:​
• Clusters or buckets​
• Partitions​
• Tables​
• Databases​

68. Explain the use of Hive in the Hadoop ecosystem.​

Hive is a data warehousing and an ETL solution for querying and analysing massive datasets
stored in the Hadoop environment. Hive has three essential purposes in Hadoop: data
summarisation, querying and analysing unstructured and semi-structured data.​
69. List various complex data types/collections supported by Hive.​
The complex data collections or types supported by Hive are as follows:​
• Array​
• Map​
• Struct​
• Union​

70. Explain how the .hiverc file in Hive is used.​

The initialisation file is called hiverc. This file is loaded when we launch the Command Line
Interface for Hive. In this file, we may specify the starting values of parameters.​
71. Is it possible to create multiple tables in Hive for a single data
file?​
Hive permits you to write data to numerous tables or folders simultaneously.​
72. Explain different SerDe implementations available in Hive.​
• For IO, Hive employs the SerDe interface. The interface supports both serialization and
deserialization, as well as serialisation results as separate fields for processing.​
• Hive can read data from a table and also write it back to the Hadoop Distributed File System
in any personalised format using a SerDe. Any individual with a computer may create their
SerDe for their data types.​
73. List table-generating functions available in Hive.​
The table-generating functions that are available in Hive are as follows:​
• explode(ARRAY)​
• explode(MAP)​
• inline(ARRAY<STRUCT[,STRUCT]>)​
• explode(array a)​
• json_tuple(jsonStr, k1, k2, …)​
• parse_url_tuple(url, p1, p2, …)​
• posexplode(ARRAY)​
• stack(INT n, v_1, v_2, …, v_k)​

74. What is a Skewed table in Hive?​

A Skewed table in Hive has values in considerable quantities compared to other data. The Skew
data is kept in a separate file, while the remainder is kept in another.​
75. List objects created by CREATE statements in MySQL.​
Using the CREATE statement, the following objects are created:​
• EVENT​
• DATABASE​
• VIEW​
• USER​
• TRIGGER​
• TABLE​
• INDEX​
• FUNCTION​
• PROCEDURE​

76. How to see the database structure in MySQL?​

To display the database structure and its properties in MySQL, you need to use the DESCRIBE
function:
DESCRIBE table_name; OR DESC table_name;.​
77. How to search for a specific String in the MySQL table column?​
The location of the first occurrence of a string within a string is returned by MySQL LOCATE().
These strings are both supplied as arguments. An optional parameter can be used to determine
where the search should begin in the string (i.e. the text to be searched).​
78. Explain how data analytics and big data can increase company
revenue.​
Big data analytics enables businesses to develop new goods based on consumer demands and
preferences. Because these things help organisations to generate more money, firms are turning
to big data analytics. Big data analytics may help businesses raise their income by 5-20%.
Furthermore, it allows businesses to understand their competitors better.​