Python Data Science Cookbook: Practical solutions across fast data cleaning, processing, and machine learning workflows with pandas, NumPy, and scikit-learn

Python Data Science Cookbook

Practical solutions across fast data cleaning, processing, and machine learning workflows with pandas, NumPy, and scikit-learn

Taryn Voska

Preface

This book's got a bunch of handy recipes for data science pros to get them through the most common challenges they face when using Python tools and libraries. Instead of going over the basics, each recipe shows you exactly how to do something step-by-step. You can load CSVs directly from a URL, flatten nested JSON, query SQL and NoSQL databases, import Excel sheets, or stream large files in memory-safe batches. That way, you spend less time on setup and more time on analysis.

Once the data's loaded, you'll find simple ways to spot and fill in missing values, standardize categories that are off, clip outliers, normalize features, get rid of duplicates, and extract the year, month, or weekday from timestamps. You'll learn how to run quick analyses, like generating descriptive statistics, plotting histograms and correlation heatmaps, building pivot tables, creating scatter-matrix plots, and drawing time-series line charts to spot trends. You'll learn how to build polynomial features, compare MinMax, Standard, and Robust scaling, smooth data with rolling averages, apply PCA to reduce dimensions, and encode high-cardinality fields with sparse one-hot encoding using feature engineering recipes.

As for machine learning, you'll learn to put together end-to-end pipelines that handle imputation, scaling, feature selection, and modeling in one object, create custom transformers, automate hyperparameter searches with GridSearchCV, save and load your pipelines, and let SelectKBest pick the top features automatically. You'll learn how to test hypotheses with t-tests and chi-square tests, build linear and Ridge regressions, work with decision trees and random forests, segment countries using clustering, and evaluate models using MSE, classification reports, and ROC curves. And you'll finally get a handle on debugging and integration: fixing pandas merge errors, correcting NumPy broadcasting mismatches, and making sure your plots are consistent.

In this book:

You can load remote CSVs directly into pandas using read_csv, so you don't have to deal with manual downloads and file clutter.

Use json_normalize to convert nested JSON responses into simple tables, making it a breeze to analyze.

You can query relational and NoSQL databases directly from Python, and the results will merge seamlessly into Pandas.

Find and fill in missing values using IGNSA(), forward-fill, and median strategies for all of your data over time.

You can free up a lot of memory by turning string columns into Pandas' Categorical dtype.

You can speed up computations with NumPy vectorization and chunked CSV reading to prevent RAM exhaustion.

You can build feature pipelines using custom transformers, scaling, and automated hyperparameter tuning with GridSearchCV.

Use regression, tree-based, and clustering algorithms to show linear, nonlinear, and group-specific vaccination patterns.

Evaluate models using MSE, R², precision, recall, and ROC curves to assess their performance.

Set up automated data retrieval with scheduled API pulls, cloud storage, Kafka streams, and GraphQL queries.

Prologue

I've been looking at a lot of job postings lately that are looking for data science professionals who can turn huge sets of data into useful info. It seems like every recruiter is looking for someone who knows their way around Python's ever-expanding ecosystem, like pandas, NumPy, scikit-learn, matplotlib, TensorFlow, and more. I get that a lot of people like the advanced features, but I also see a lot of frustration when people have to juggle dozens of libraries just to get something done. As Python Data Science Cookbook takes shape, I'm aiming to share clear, hands-on solutions that'll help you work quickly and confidently, without getting bogged down by complexity.

When I was just starting out, I had the same problems. I spent days searching for the right function to flatten a nested JSON or struggling with inconsistent column names when merging datasets. I watched the memory usage go up and up until my machine slowed way down. I was writing long loops in pure Python, but then I found out that NumPy had a faster and more elegant approach. I'd build ad hoc scripts, then duplicate code across projects when slight tweaks were needed. Every time, I felt a little bit of regret. I could have spent more time refining my analysis instead of wrestling with tooling problems.

I learned that practical, self-contained recipes are more valuable than huge manuals that cover every corner of a library's API. I want this book to be a reliable resource—something you can pick up when you hit a roadblock. Need to pull a CSV straight from a GitHub repository? Flick to the first recipe. Struggling with missing values? Check out the chapter on data cleaning. At each step, you'll see code fragments that you can copy, paste, and adapt. You'll also learn what to check when something goes wrong, how to inspect merge conflicts, how to fix NumPy broadcasting errors, and how to profile memory usage so leaks don't derail long-running tasks.

As you make your way through the chapters, you'll find that acquiring data can sometimes feel like the toughest part. You'll get to practice pulling data from REST APIs, consuming GraphQL endpoints, fetching metadata from MongoDB, and even scheduling automatic downloads so your local datasets stay fresh. You'll learn how to upload and retrieve files from cloud storage, like Amazon S3 and Google Cloud Storage. This way, you can work with large CSVs or model artifacts without putting too much strain on your local disk.

When you import your data into Pandas, you'll see simple ways to normalize labels, remove outliers, build features and create cool pivot tables. You'll move past static tables into visualizations like histograms, heatmaps, scatter-matrix plots, and time-series line charts. These visuals will help you spot trends, clusters, and outliers in just a few lines of code. You'll also learn how to optimize for speed and memory, like converting text columns to categorical types, reading CSVs in chunks, memory-mapping large arrays, and setting indices for rapid joins.

Finally, you'll get to play with statistical tests and machine learning methods like t-tests, chi-square tests, linear and Ridge regression, decision trees, random forests, and clustering. And you'll learn how to evaluate models with mean squared error, R², precision, recall, and ROC curves. You'll put together pipelines that handle imputation, scaling, feature selection, and modeling all in one object, and then save those pipelines so you can use them again.

I wrote this cookbook to save you time troubleshooting and more time discovering insights. These recipes tackle the literal problems you'll face—mismatched keys, shape errors, memory leaks, rate limits—so that each step builds toward a smooth, automated workflow.

--Taryn Voska

Copyright © 2024 by GitforGits

All rights reserved. This book is protected under copyright laws and no part of it may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without the prior written permission of the publisher. Any unauthorized reproduction, distribution, or transmission of this work may result in civil and criminal penalties and will be dealt with in the respective jurisdiction at anywhere in India, in accordance with the applicable copyright laws.

Published by: GitforGits

Publisher: Sonal Dhandre

www.gitforgits.com

[email protected]

Printed in India

First Printing: February 2025

Cover Design by: Kitten Publishing

For permission to use material from