Machine Learning With Python Programming : 2023 A Beginners Guide

© Copyrıght 2023 by ORCHID PUBLISHING- All rıghts reserved.

Thıs document ıs geared towards provıdıng exact and relıable ınformatıon ın regards to the topıc and ıssue covered. The publıcatıon ıs sold wıth the ıdea that the publısher ıs not requıred to render accountıng, offıcıally permıtted, or otherwıse, qualıfıed servıces. If advıce ıs necessary, legal or professıonal, a practıced ındıvıdual ın the professıon should be ordered.

- From a Declaratıon of Prıncıples whıch was accepted and approved equally by a Commıttee of the Amerıcan Bar Assocıatıon and a Commıttee of Publıshers and Assocıatıons.

In no way ıs ıt legal to reproduce, duplıcate, or transmıt any part of thıs document ın eıther electronıc means or ın prınted format. Recordıng of thıs publıcatıon ıs strıctly prohıbıted and any storage of thıs document ıs not allowed unless wıth wrıtten permıssıon from the publısher. All rıghts reserved.

The ınformatıon provıded hereın ıs stated to be truthful and consıstent, ın that any lıabılıty, ın terms of ınattentıon or otherwıse, by any usage or abuse of any polıcıes, processes, or dırectıons contaıned wıthın ıs the solıtary and utter responsıbılıty of the recıpıent reader. Under no cırcumstances wıll any legal responsıbılıty or blame be held agaınst the publısher for any reparatıon, damages, or monetary loss due to the ınformatıon hereın, eıther dırectly or ındırectly.

Respectıve authors own all copyrıghts not held by the publısher.

The ınformatıon hereın ıs offered for ınformatıonal purposes solely, and ıs unıversal as so. The presentatıon of the ınformatıon ıs wıthout contract or any type of guarantee assurance.

The trademarks that are used are wıthout any consent, and the publıcatıon of the trademark ıs wıthout permıssıon or backıng by the trademark owner.

All trademarks and brands wıthın thıs book are for clarıfyıng purposes only and are the owned by the owners themselves, not affılıated wıth thıs document.

Table of Contents

© Copyrıght 2023 by ORCHID PUBLISHING- All rıghts reserved...................3

Table of Contents...................................................................4

About Author...................................................................13

Who This Book Is For..........................................................14

INTRODUCTION.................................................................15

Chapter 1 Overview of Artificial Intelligence.............................16

A basic supervised model: Neighbor to Neighbor....................................3

Changing Hyperparameters using Cross-Validation................................11

1.2 Preprocessing..................................................................16

1.2.1 Scaling Data..................................................................15

1.2 Techniques for Handling Unbalanced Information............................26

1.3 Reducing Dimensionality: Principal Component Analysis.....................31

Chapter 2 Python Machine Learning Ecosystem.........................45

Python.............................................................................45

SciPy...............................................................................46

scikit-learn.........................................................................46

Python Installation of Ecosystems..................................................11

Installing Python: A Guide..........................................................11

Ways To Set Up SciPy..............................................................11

Installing Scikit-Learn: A Guide......................................................12

An Easy Method for Installing the Ecosystem.........................................13

Summary.........................................................................14

Next..............................................................................14

Chapter 3 A Quick Course on SciPy and Python.........................15

Crash Course in Python.............................................................15

Assignment.......................................................................16

Flow Control.......................................................................17

Data Structures...................................................................18

3.1 NumPy Crash Course....................................................20

3.1.1 Create Array.................................................................20

3.1.2 Access Data.................................................................21

3.1.3 Arithmetic...................................................................21

3.2 Matplotlib Crash Course................................................22

3.2.1 Line Plot.....................................................................22

3.2.2 Scatter Plot.................................................................23

3.3 Pandas Crash Course....................................................24

3.3.1 Series.......................................................................24

3.3.2 DataFrame..................................................................25

3.4 Summary...................................................................26

3.4.1 Next.........................................................................26

Chapter 4 How to Import Data for Machine Learning..................27

4.1 Considerations When Loading CSV Data............................27

4.1.1 File Header..................................................................27

4.1.2 Comments...................................................................27

4.1.3 Delimiter....................................................................28

4.1.4 Quotes......................................................................28

4.2 Pima Indians Dataset....................................................28

4.3 Load CSV Files with the Python Standard Library.................28

4.4 Load CSV Files with NumPy............................................29

4.5 Load CSV Files with Pandas............................................29

Summary...........................................................................29

Next..............................................................................30

Chapter 5  Use Descriptive Statistics to Gain Understanding of Your Data  31

5.1 Peek at Your Data........................................................31

5.2 Dimensions of Your Data...............................................32

5.3 Data Type For Each Attribute..........................................33

5.4 Descriptive Statistics....................................................33

5.5 Class Distribution (Classification Only)..............................34

5.6 Correlations Between Attributes......................................35

5.7 Skew of Univariate Distributions......................................36

5.9 Summary...................................................................37

5.9.1 Next.........................................................................37

Chapter 6 Understand Your Data With Visualization...................38

6.1 Univariate Plots..........................................................38

6.1.1 Histograms..................................................................38

6.1.2 Density Plots................................................................39

6.1.3 Box and Whisker Plots......................................................40

6.2 Multivariate Plots........................................................41

6.2.1 Correlation Matrix Plot......................................................41

6.2.2 Scatter Plot Matrix..........................................................44

6.3 Summary...................................................................45

6.3.1 Next.........................................................................46

Chapter 7 Get Ready for Machine Learning with Your Data...........47

7.1 Requirement for Pre-processing Data...............................47

7.2 Data Transforms..........................................................47

7.3 Rescale Data..............................................................48

7.4 Standardize Data.........................................................49

7.5 Normalize Data...........................................................50

7.6 Binarize Data (Make Binary)...........................................50

7.7 Summary...................................................................51

7.7.1 Next.........................................................................51

Chapter 8 Choosing Features for Machine Learning....................52

8.1 Feature Selection........................................................52

8.2 Univariate Selection.....................................................53

8.3 Recursive Feature Elimination.........................................53

8.4 Principal Component Analysis.........................................54

8.5 Feature Importance......................................................55

8.6 Summary...................................................................56

8.6.1 Next.........................................................................56

Chapter 9 Analyze Machine Learning Algorithms' Performance Using Resampling  57

9.1 Assess Algorithms for Machine Learning............................57

9.2 Divided Into Test and Train Sets......................................58

9.3 K-fold Cross Validation..................................................59

9.4 Departure-One Cross-Validation......................................60

9.5 Continual Random Test-Train Splitting..............................62

9.6 Which Methods to Apply When........................................63

9.7 Summary...................................................................63

9.7.1 Next.........................................................................63

Chapter 10 Performance Measures for Algorithms in Machine Learning  64

10.1 Metrics for Algorithm Evaluation............................................64

10.2 Measures of Classification......................................................65

Chapter 11 Spot-Check Classification Algorithms.......................70

Algorithm Spot-Checking...........................................................70

Algorithms Overview...............................................................71

Linear Machine Learning Algorithms...............................................71

Logistic Regression................................................................71

Linear Discriminant Analysis........................................................72

Nonlinear Machine Learning Algorithms............................................72

k-Nearest Neighbors...............................................................73

Naive Bayes.......................................................................73

Classification and Regression Trees.................................................74

Support Vector Machines...........................................................74

11.1 Summary...................................................................75

11.1.1 Next....................................................................75

Chapter 12 Algorithms for Spot-Check Regression.....................76

12.1 Algorithms Overview....................................................76

12.2 Linear Machine Learning Algorithms.................................77

12.2.1 Linear Regression.......................................................77

12.2.2 Ridge Regression.......................................................78

12.2.3 LASSO Regression......................................................78

12.2.4 ElasticNet Regression...................................................79

12.3 Nonlinear Machine Learning Algorithms............................80

12.3.1 K-Nearest Neighbors....................................................81

12.3.2 Classification and Regression Trees....................................81

12.3.3 Support Vector Machines...............................................82

12.4 Summary...................................................................83

12.4.1 Next....................................................................83

Chapter 13 Compare Machine Learning Algorithms....................84

13.1 Selecting The Optimal Machine Learning Model...................84

13.2 Regularly Compare Algorithms for Machine Learning............84

13.3 Summary...................................................................86

13.3.1 Next....................................................................86

Chapter 14 Use Pipelines to Automate Machine Learning Workflows87

14.1 Automating Processes for Machine Learning.......................87

14.2 Pipeline for Preparing Data and Modeling..........................87

14.3 Pipeline for Feature Extraction and Modeling......................89

14.4 Summary...................................................................90

14.4.1 Next....................................................................90

Chapter 15 Boost Performance in Group Settings......................91

15.1 Create ensemble predictions by combining models...............91

15.2 Bagging  Algorithms.....................................................92

15.2.1 Bagged Decision Trees..................................................92

15.2.2 Random Forest..........................................................93

15.2.3 Extra Trees..............................................................93

15.3 Boosting Algorithms.....................................................94

15.3.1 AdaBoost...............................................................94

15.3.2 Stochastic Gradient Boosting...........................................95

15.4 Voting Ensemble.........................................................96

15.5 Summary...................................................................97

15.5.1 Next....................................................................97

Chapter 16 Boost Efficiency via Algorithm Adjustment................98

16.1 Parameters of Machine Learning Algorithms.......................98

16.2 Adjusting the Grid Search Parameter................................98

16.3 Adjusting the Random Search Parameter...........................99

16.4 Summary..................................................................100

16.4.1 Next...................................................................100

Chapter 17 Store and Import Deep Learning Models..................101

17.1 Use Pickle to Complete Your Model.................................101

17.2 Finalize Your Model with Joblib......................................102

17.3 Advice on Concluding Your Model...................................103

17.4 Summary..................................................................103

17.4.1 Next...................................................................104

Chapter 18 Template for Predictive Modeling Projects...............105

18.1 Use Projects to Practice Machine Learning........................105

18.1.1 Employ a Methodical, Structured Approach............................105

18.2 Machine Learning Project Template in Python....................106

18.2.1 Template Summary....................................................106

18.2.2 How To Use The Project Template.....................................108

18.3 Machine Learning Project Template Steps.........................108

18.3.1 Prepare Problem.......................................................108

18.3.2 Summarize Data.......................................................108

18.3.3 Prepare Data...........................................................108

18.3.4 Evaluate Algorithms...................................................109

18.3.5 Improve Accuracy......................................................109

18.3.6 Finalize Model.........................................................109

18.4 Tips For Using The Template Well...................................110

18.5 Summary..................................................................110

18.5.1 Next Step..............................................................110

Chapter 19 Your First Machine Learning Project in Python Step-By-Step  111

19.1 The Hello World of Machine Learning...............................111

19.2 Load The Data...........................................................112

19.2.1 Import libraries........................................................112

19.2.2 Load Dataset...........................................................112

19.3 Summarize the Dataset................................................113

19.3.1 Dimensions of Dataset.................................................113

19.3.2 Peek at the Data.......................................................113

19.3.3 Statistical Summary...................................................114

19.3.4 Class Distribution......................................................115

19.4 Data Visualization......................................................116

19.4.1 Univariate Plots........................................................116

19.4.2 Multivariate Plots......................................................118

19.5 Evaluate Some Algorithms............................................119

19.5.1 Create a Validation Dataset............................................119

19.5.2 Test Harness...........................................................120

19.5.3 Build Models...........................................................120

19.5.4 Select The Best Model.................................................121

19.6 Make Predictions........................................................122

19.7 Summary..................................................................123

19.7.1 Next Step..............................................................123

Chapter 20 Regression Machine Learning Case Study Project......124

20.1 Problem Definition......................................................124

20.2 Load the Dataset........................................................125

20.3 Analyze Data.............................................................125

20.3.1 Descriptive Statistics..................................................125

20.4 Data Visualizations.....................................................128

20.4.1 Unimodal Data Visualizations..........................................128

20.4.2 Multimodal Data Visualizations........................................131

20.4.3 Summary of Ideas......................................................133

20.5 Validation Dataset......................................................133

20.6 Evaluate Algorithms: Baseline.......................................134

20.7 Evaluate Algorithms: Standardization..............................136

20.8 Improve Results With Tuning.........................................138

20.9 Ensemble  Methods.....................................................139

20.10 Tune Ensemble Methods.........................................141

20.11 Finalize Model.....................................................142

20.12 Summary............................................................143

20.12.1  Next Step................................................................143

Chapter 21 Binary Classification Machine Learning Case Study Project  144

21.1  Problem Definition....................................................144

21.2 Load the Dataset........................................................144

21.3 Analyze Data.............................................................145

21.3.1 Descriptive Statistics..................................................145

21.3.2 Unimodal Data Visualizations..........................................149

21.3.3 Multimodal Data Visualizations........................................152

21.4 Validation Dataset......................................................153

21.5 Evaluate Algorithms: Baseline.......................................154

21.6 Evaluate Algorithms: Standardize Data............................156

21.7 Algorithm Tuning........................................................158

21.7.1 Tuning KNN............................................................158

21.7.2 Tuning SVM............................................................159

21.8 Ensemble Methods......................................................162

21.9 Finalize Model...........................................................163

21.10 Summary............................................................164

21.10.1 Next Step..............................................................164

Chapter 22 More Predictive Modeling Projects.........................165

22.1 Build And Maintain Recipes...........................................165

22.2 Small Projects on Small Datasets....................................165

22.3 Competitive Machine Learning.......................................166

22.4 Summary..................................................................166

About Author

––––––––

Dr. James Harrison holds bachelor’s and master’s degrees in mechanical engineering, an ScD in instrumentation, and an MBA. He has worked in aca- demia, technology, and business. Mike currently works with companies where artificial intelligence or machine learning are integral to success. He serves var- iously as part of the management team, a consultant, or advisor. He also teaches machine learning courses at UC Berkeley and Hacker Dojo, a co-working space and startup incubator in Mountain View, CA.

James was born in Oklahoma and took his bachelor’s and master’s degrees there, then after a stint in Southeast Asia went to Cambridge for ScD and C. Stark Draper Chair at MIT after graduation. James left Boston to work on com- munications satellites at Hughes Aircraft Company in Southern California, and then after completing an MBA at UCLA moved to the San Francisco Bay Area to take roles as founder and CEO of two successful venture-backed startups.

James remains actively involved in technical and startup-related work. Recent projects include the use of machine learning in industrial inspection and auto- mation, financial prediction, predicting biological outcomes on the basis of molecular graph structures, and financial risk estimation. He has participated in due diligence work on companies in the artificial intelligence and machine learning arenas. James can be reached through mbowles.com.

Who This Book Is For

This book is intended for Python programmers who want to add machine learning to their repertoire, either for a specific project or as part of keeping their toolkit relevant. Perhaps a new problem has come up at work that requires machine learning. With machine learning being covered so much in the news these days, it’s a useful skill to claim on a resume.

This book provides the following for Python programmers:

A description of the basic problems that machine learning attacks

Several state-of-the-art algorithms

The principles of operation for these algorithms

Process steps for specifying, designing, and qualifying a machine learning system

Examples of the processes and algorithms

Hackable code

To get through this book easily, your primary background requirements include an understanding of programming or computer science and the ability to read and write code. The code examples, libraries, and packages are all Python, so the book will prove most useful to Python programmers. In some cases, the book runs through code for the core of an algorithm to demonstrate the operating principles, but then uses a Python package incorporating the algorithm to apply the algorithm to problems. Seeing code often gives programmers an intuitive grasp of an algorithm in the way that seeing the math does for others. Once the understanding is in place, examples will use developed Python packages with the bells and whistles that are important for efficient use (error checking, handling input and output, developed data structures for the models, defined predictor methods incorporating the trained model, and so on).

In addition to having a programming background, some knowledge of math and statistics will help get you through the material easily. Math requirements include some undergraduate-level differential calculus (knowing how to take a derivative and a little bit of linear algebra), matrix notation, matrix multiplication, and matrix inverse. The main use of these will be to follow the derivations of some of the algorithms covered. Many times, that will be as simple as taking a derivative of a simple function or doing some basic matrix manipulations. Being able to follow the calculations at a conceptual level may aid your understanding of the algorithm. Understanding the steps in the derivation can help you to under- stand the strengths and weaknesses of an algorithm and can help you to decide which algorithm is likely to be the best choice for a particular problem.

INTRODUCTION

Extracting actionable information from data is changing the fabric of modern business in ways that directly affect programmers. One way is the demand for new programming skills. Market analysts predict demand for people with advanced statistics and machine learning skills will exceed supply by 140,000 to 190,000 by 2018. That means good salaries and a wide choice of interesting projects for those who have the requisite skills. Another development that affects programmers is progress in developing core tools for statistics and machine learning. This relieves programmers of the need to program intricate algorithms for themselves each time they want to try a new one. Among general-purpose programming languages, Python developers have been in the forefront, building state-of-the-art machine learning tools, but there is a gap between having the tools and being able to use them efficiently.

Programmers can gain general knowledge about machine learning in a number of ways: online courses, a number of well-written books, and so on. Many of these give excellent surveys of machine learning algorithms and examples of their use, but because of the availability of so many different algorithms, it’s difficult to cover the details of their usage in a survey.

This leaves a gap for the practitioner. The number of algorithms available requires making choices that a programmer new to machine learning might not be equipped to make until trying several, and it leaves the programmer to fill in the details of the usage of these algorithms in the context of overall problem formulation and solution.

This book attempts to close that gap. The approach taken is to restrict the algo- rithms covered to two families of algorithms that have proven to give optimum performance for a wide variety of problems. This assertion is supported by their dominant usage in machine learning competitions, their early inclusion in

newly developed packages of machine learning tools, and their performance in comparative studies (as discussed in Chapter 1, The Two Essential Algorithms for Making Predictions). Restricting attention to two algorithm families makes it possible to provide good coverage of the principles of operation and to run through the details of a number of examples showing how these algorithms apply to problems with different structures.

The book largely relies on code examples to illustrate the principles of oper- ation for the algorithms discussed. I’ve discovered in the classes I have taught at University of California, Berkeley, Galvanize, University of New Haven, and Hacker Dojo, that programmers generally grasp principles more readily by seeing simple code illustrations than by looking at math.

This book focuses on