Business Intelligence and Data Mining Techniques

Business Intelligence and Data Mining Techniques

Business Intelligence and Data Mining Techniques

By

Dwaipayan Sethi

Business Intelligence and Data Mining Techniques

Dwaipayan Sethi

ISBN - 9789361527081

COPYRIGHT © 2025 by Educohack Press. All rights reserved.

This work is protected by copyright, and all rights are reserved by the Publisher. This includes, but is not limited to, the rights to translate, reprint, reproduce, broadcast, electronically store or retrieve, and adapt the work using any methodology, whether currently known or developed in the future.

The use of general descriptive names, registered names, trademarks, service marks, or similar designations in this publication does not imply that such terms are exempt from applicable protective laws and regulations or that they are available for unrestricted use.

The Publisher, authors, and editors have taken great care to ensure the accuracy and reliability of the information presented in this publication at the time of its release. However, no explicit or implied guarantees are provided regarding the accuracy, completeness, or suitability of the content for any particular purpose.

If you identify any errors or omissions, please notify us promptly at [email protected] & [email protected] We deeply value your feedback and will take appropriate corrective actions.

The Publisher remains neutral concerning jurisdictional claims in published maps and institutional affiliations.

Published by Educohack Press, House No. 537, Delhi- 110042, INDIA

Email: [email protected] & [email protected]

Cover design by Team EDUCOHACK

Preface

Data is the proverbial blood that powers the 21st century corporate economy. The mere mention may spawn fanciful scenarios, but in reality data holds the key to unlocking human productivity in all areas. Climate change, business failure, epidemics and crop production can be understood with the right data insights. Data availability shortens our learning curve in problem solving.

As important as finding the right product market fit for a company is data mining for sustainable, self-sustaining enterprise business intelligence. It helps future roadmaps, product development, and the many business processes that keep the winning wheels spinning.

Therefore, this book clearly explains the topics related to data mining and business intelligence, the importance of data mining, and how to perform data mining to ensure a seamless revenue stream.

The importance of data mining in business is that it is used to transform raw data into meaningful, consumable and actionable insights.

Data mining refers to the process of extracting information from large datasets and data analysis is the process used to find patterns in the extracted information. Data analysis includes phases such as data exploration, cleaning, transformation, and modelling. The goal of this book is to find information, draw conclusions, and act on them.

Table of Contents

Chapter 1

Wholeness of Business Intelligence and Data Mining 1

1.1 Business Intelligence 1

1.2 Pattern Recognition 2

1.3 Data Processing Chain 4

Quick Recap 10

Questionnaire 10

References 11

Chapter 2

Business Intelligence Concepts and its Applications 13

2.1 BI for Better Decisions 13

2.2 Decision Types 14

2.3 BI Tools 15

2.4 BI Skills 15

2.5 BI Applications 16

Quick Recap 24

Questionnaire 25

References 25

Chapter 3

Data Warehousing 27

3.1 Design Considerations for DW 27

3.2 DW development Approaches 28

3.3 DW Architecture 29

3.4 Data Sources 29

3.5 Data Loading Processes 29

3.6 DW Design 29

3.7 DW Access 30

3.8 DW Best Practices 30

Quick Recap 31

Questionnaire 32

References 32

Chapter 4

Data Mining 33

4.1 Gathering and Selecting Data 33

4.2 Data Cleansing and Preparation 35

4.3 Outputs of Data Mining 36

4.4 Evaluating Data Mining Results 37

4.5 Data Mining Techniques 37

4.6 Tools and Platforms for Data Mining 39

4.7 Data Mining Best Practices 40

4.8 Myths about Data Mining 41

4.9 Data Mining Mistakes 42

Quick Recap 43

Questionnaire 44

References 44

Chapter 5

Decision Trees 47

5.1 Decision Tree Problem 47

5.2 Decision Tree Construction 49

5.3 Lessons from Constructing Trees 50

5.4 Decision Tree Algorithms 51

Quick Recap 52

Questionnaire 54

References 54

Chapter 6

Regression 55

6.1 Correlations and Relationships 55

6.2 Visual Look At Relationships 56

6.3 Regression 57

6.4 Non Linear Regression 58

6.5 Logistic Regression 58

6.6 Advantages and Disadvantages of Regression Models 59

Quick Recap 59

Questionnaire 61

References 61

Chapter 7

Artificial Neural Networks 63

7.1 Business Applications of ANN 63

7.2 Design Principles of ANN 64

7.3 Representation of ANN 65

7.4 Architecting a Neural Network 65

7.5 Developing an ANN 65

7.6 Advantages and Disadvantages of using ANN 66

Quick Recap 67

Questionnaire 68

References 68

Chapter 8

Cluster Analysis 71

8.1 Applications of Cluster Analysis 71

8.2 Definition of a Cluster 72

8.3 Representing Clusters 72

8.4 Clustering Techniques 73

8.5 K-means Algorithm For Clustering 73

8.6 Selecting the number of Clusters 74

8.7 Advantages and Disadvantages of K-means Algorithm 74

Quick Recap 75

Questionnaire 76

References 76

Chapter 9

Association Rule Mining 79

9.1 Business Applications of Association Rules 79

9.2 Representing Association Rules 80

9.3 Algorithms for Association Rules 80

9.4 Apriori Algorithm 80

9.5 Creating Association Rules 81

Quick Recap 82

Questionnaire 83

References 83

Chapter 10

Text Mining 85

10.1 Text Mining Applications 85

10.2 Text Mining Process 86

10.3 Mining the TDM 88

10.4 Comparing Text Mining and Data Mining 89

10.5 Text Mining Best Practices 89

Quick Recap 90

Questionnaire 91

References 91

Chapter 11

Web Mining 93

11.1 Web Content Mining 93

11.2 Web Structure Mining 94

11.3 Web Usage Mining 94

11.4 Web Mining Algorithms 95

Quick Recap 96

Questionnaire 97

References 97

Chapter 12

Big Data 99

12.1 Definition of Big Data 99

12.2 Big Data Landscape 101

12.3 Business Implications of Big Data 101

12.4 Technology Implications of Big Data 102

12.5 Big Data Technologies 103

12.6 Management of Big Data 103

Quick Recap 104

Questionnaire 105

References 105

Chapter 13

Data Modelling Primer 107

13.1 Data Modelling Primer 107

13.2 Evolution of Data Management Systems 108

13.3 Relational Data Models 109

13.4 Implementing Relational Data Models 111

13.5 DBMS 111

Quick Recap 111

Questionnaire 112

References 113

Chapter 14

DM and BI: How do they Work Together? 115

14.1 Introduction 115

14.2 BI 116

14.3 Data Mining 117

14.4 Data Preparation 118

14.5 Data Manipulation 119

14.6 Data Manipulation Language 119

14.7 Data Mining VS Business Intelligence 120

14.8 Data Mining and Business Intelligence: How They Work Together 120

Quick Recap 121

Questionnaire 122

References 122

Chapter 15

Data Profiling 125

15.1 Introduction 125

15.2 Types of Data Profiling 126

15.3 Steps in Data Profiling Process 126

15.4 Benefits 127

15.5 Challenges 127

15.6 Examples 128

15.7 Tools 128

Quick Recap 129

Questionnaire 130

References 130

Chapter 16

How to Pair Data Mining and Business Intelligence in 2023 133

16.1 BI 133

16.2 How does data mining help business intelligence? 134

16.3 What are the benefits of data mining in business intelligence? 134

16.4 What are the challenges of data mining in business intelligence? 135

16.5 Which industries benefit the most from data mining in business intelligence? 136

Quick Recap 137

Questionnaire 137

References 137

Glossary 139

Index 143

Chapter 1 Wholeness of Business Intelligence and Data Mining

1.1 Business Intelligence

All business organisations need to continuously monitor the business environment and their own performance and quickly adjust their plans for the future. This includes industry, competitor, supplier and customer monitoring. Organisations should also create a Balanced Scorecard to track their health and vitality.

Executives typically decide what to track based on Key Performance Indicators (KPIs) or Key Result Areas (KRAs). Customised reports should be created to provide each manager with the information they need. These reports can be transformed into custom dashboards that provide information quickly and in an easy-to-understand format.

Business intelligence is a broad range of information technology (IT) solutions that include tools for gathering, analysing, and reporting to users information about the performance of an organisation and its environment. These IT solutions are among the top investment solutions.

Imagine a retail chain that sells a wide variety of goods and services from all over the world online and in physical stores. Generate data on sales, purchases and expenses from multiple locations and timeframes.

Fig 1.1 Business Intelligence

https://www.slideteam.net/media/catalog/product/cache/1280x720/f/i/five_steps_business_intelligence_process_with_data_mining_pyramid_Slide01.jpg

Analysing this data can help you identify top-selling products, locally-sold products, seasonal products, fast-growing customer segments, and more. It also helps generate ideas about which products are sold together, which users tend to buy which products, and so on. These insights and information help design better advertising plans, product packaging and store layouts, which in turn lead to better business development.

A retail company’s vice president of sales wants to track sales history based on monthly goals, the performance of each store and product category, and the top store leaders for the month. The VP of Finance is interested in tracking her daily income, expenses, and cash flow by branch. Compare them with your plans. measurement of cost of capital; etc.

1.2 Pattern Recognition

A pattern is a design or model that helps you capture something. Patterns help connect things that seem unconnected. Patterns help break through complexity and show trends that are easier to understand. Patterns can be as clear as rigid scientific rules, like the rule that the sun always rises in the east. It can also be a simple generalisation, such as Pareto’s Principle that 80% of effects come from 20% of causes. A complete pattern or model is one that

(a) describes the situation precisely,

(b) is broadly applicable, and

(c) can be described in a simple way.

Fig 1.2 BI VS DM

https://static.javatpoint.com/tutorial/data-mining/images/business-intelligence-vs-data-mining.png

E = MC² would be such a general, accurate and simple (GAS) model. Often you can’t achieve all three qualities in one model, and you need to be happy with two out of the three qualities of your model.

Patterns can be temporary and occur regularly over time. Patterns can also be spatial. B. Organized in a particular way. Patterns can be functional in that certain things lead to certain effects. Good patterns are often symmetrical. They reflect basic structures and patterns that we already know.

The timing rule is that some people will always be late, regardless of the season or time. Some are familiar with this pattern, others are not. Understanding these patterns can help diffuse a lot of unnecessary frustration and anger. It’s easy to joke and laugh that some people were born ten minutes late. Similarly, according to Parkinson’s Law, work expands to fill all the time available for it.

In a spatial pattern that follows the 80-20 rule, the top 20% customers are likely to drive 80% of the business. Or 20% of the product is his 80% of the business. Or 80% of calls to customer service are for only 20% of products. This last pattern may simply reveal a discrepancy between a product’s attributes and what customers believe about it. Businesses can then decide to invest in better customer education and significantly reduce customer service calls.

Fig 1.3 BI

https://img.freepik.com/premium-vector/business-intelligence-steps-business-plan-data-mining-analysis-strategy_35632-173.jpg?w=2000

Functional patterns can include test functionality. Some students do well with essay type questions. Others do well with multiple choice questions. In addition, some students excel in practical projects and oral presentations. Recognizing such patterns in a class of students can help a teacher design her mechanisms of balanced testing that are fair to all.

Retaining students is a constant challenge for universities. Recent data-driven research shows that students are more likely to leave school for social reasons than for academic reasons. This pattern/insight can encourage schools to pay more attention to students engaging in extracurricular activities and build stronger bonds at school. may invest. Schools can also begin to proactively collect data on each student’s participation in these activities in order to predict which students are at risk and to take corrective action.

However, patterns established over the years can also be broken. The past cannot always predict the future. The pattern all swans are white does not mean that there are no black swans. When enough anomalies are found, the underlying pattern can change. The economic collapse of 2008 and 2009 was caused by the collapse of the common sense that house prices will always rise. The deregulated financial environment has made markets more volatile, leading to large market fluctuations and ultimately to the collapse of the entire financial system.

Diamond mining involves excavating large amounts of unrefined ore to discover precious gemstones and nuggets. Similarly, data mining involves sifting through large amounts of raw data to discover unique, important and useful patterns. After the data is cleaned up, we can apply special tools and techniques to look for patterns. By immersing yourself in clean data organised in the right perspective, you are more likely to make the right discoveries.

Experienced diamond prospectors know what diamonds look like. Similarly, an experienced data miner should know what kinds of patterns to look for. Patterns are essentially about what is connected and what is isolated. Therefore, it is very important to know your business domain well.

Discovering patterns requires knowledge and skill. It’s like looking for a needle in a haystack. Sometimes the pattern is hidden where you can’t see it. Sometimes it takes a lot of work and extensive searching to find surprisingly useful patterns. Therefore, a systematic approach to data mining is required to efficiently extract valuable insights.

Fig 1.4 Identify the data

https://acn-marketing-blog.accenture.com/wp-content/uploads/2020/03/web-data-sources-1024x682.png

For example, we can assume that an employee’s attitude towards their employer is determined by various factors such as education level, income, seniority, and gender. You might be surprised if the data show that attitudes are largely determined by age group. Simple insights like this can be a powerful force in effectively designing your organisation. Data miners must be open to all possibilities.

Done well, data mining can yield interesting insights and be a source of new ideas and initiatives. The movement of a mobile phone (inside a car) on a highway can predict highway traffic patterns. If your phone’s location isn’t moving fast enough on a highway or driveway, it could be a sign of a traffic jam. Telecom companies can therefore provide drivers with real-time traffic information on mobile phones and GPS devices without the need for video cameras or traffic reporters.

Similarly, an organisation can know what time an employee arrives at the office when their mobile phone is found in a parking lot. Observing the records of parking permit card swipes in company parking lots can let organisations know when an employee is in or out of the office at any given time.

Some patterns are very sparse and require very large amounts of disparate data to be viewed together to see connectivity. For example, finding the wreckage of an aeroplane that may have disappeared mid-course requires gathering data from many sources, including satellites, ships, and navigation systems. The quality of raw data can vary and even be inconsistent. You may or may not have enough data to find good patterns. To resolve this issue, you may need to add additional data dimensions.

Fig 1.5 Business world

https://d3i71xaburhd42.cloudfront.net/337b3351a96e692e37ead5ec90e11ba811b74fda/8-Figure2-1.png

1.3 Data Processing Chain

Data is the new natural resource. Implicit in this statement is the recognition of the hidden value of data. Data is at the heart of business intelligence. There are a number of steps that must be followed in order to systematically benefit from data. Data can be modelled and stored in a database. Relevant data can