Explore the full ebook collection and download it now at textbookfull.

com

Learn PySpark: Build python-based machine learning

and deep learning models 1st Edition Pramod Singh

https://textbookfull.com/product/learn-pyspark-build-python-
based-machine-learning-and-deep-learning-models-1st-edition-
pramod-singh/

OR CLICK HERE

DOWLOAD EBOOK

Browse and Get More Ebook Downloads Instantly at https://textbookfull.com

Click here to visit textbookfull.com and download textbook now
 Your digital treasures (PDF, ePub, MOBI) await
Download instantly and pick your perfect format...

Read anywhere, anytime, on any device!

Machine Learning with PySpark: With Natural Language

Processing and Recommender Systems 1st Edition Pramod
Singh
https://textbookfull.com/product/machine-learning-with-pyspark-with-
natural-language-processing-and-recommender-systems-1st-edition-
pramod-singh/
textbookfull.com

Deep Learning with Python Learn Best Practices of Deep

Learning Models with PyTorch 2nd Edition Nikhil Ketkar
Jojo Moolayil
https://textbookfull.com/product/deep-learning-with-python-learn-best-
practices-of-deep-learning-models-with-pytorch-2nd-edition-nikhil-
ketkar-jojo-moolayil/
textbookfull.com

Deep Learning with Python Learn Best Practices of Deep

Learning Models with PyTorch 2nd Edition Nikhil Ketkar
Jojo Moolayil
https://textbookfull.com/product/deep-learning-with-python-learn-best-
practices-of-deep-learning-models-with-pytorch-2nd-edition-nikhil-
ketkar-jojo-moolayil-2/
textbookfull.com

Deploy Machine Learning Models to Production: With Flask,

Streamlit, Docker, and Kubernetes on Google Cloud Platform
Pramod Singh
https://textbookfull.com/product/deploy-machine-learning-models-to-
production-with-flask-streamlit-docker-and-kubernetes-on-google-cloud-
platform-pramod-singh/
textbookfull.com
Deploy Machine Learning Models to Production: With Flask,
Streamlit, Docker, and Kubernetes on Google Cloud Platform
1st Edition Pramod Singh
https://textbookfull.com/product/deploy-machine-learning-models-to-
production-with-flask-streamlit-docker-and-kubernetes-on-google-cloud-
platform-1st-edition-pramod-singh/
textbookfull.com

Hyperparameter Optimization in Machine Learning: Make Your

Machine Learning and Deep Learning Models More Efficient
1st Edition Tanay Agrawal
https://textbookfull.com/product/hyperparameter-optimization-in-
machine-learning-make-your-machine-learning-and-deep-learning-models-
more-efficient-1st-edition-tanay-agrawal/
textbookfull.com

Hyperparameter Optimization in Machine Learning: Make Your

Machine Learning and Deep Learning Models More Efficient
1st Edition Tanay Agrawal
https://textbookfull.com/product/hyperparameter-optimization-in-
machine-learning-make-your-machine-learning-and-deep-learning-models-
more-efficient-1st-edition-tanay-agrawal-2/
textbookfull.com

Deep Learning with Python Develop Deep Learning Models on

Theano and TensorFLow Using Keras Jason Brownlee

https://textbookfull.com/product/deep-learning-with-python-develop-
deep-learning-models-on-theano-and-tensorflow-using-keras-jason-
brownlee/
textbookfull.com

Practical Machine Learning with AWS : Process, Build,

Deploy, and Productionize Your Models Using AWS Himanshu
Singh
https://textbookfull.com/product/practical-machine-learning-with-aws-
process-build-deploy-and-productionize-your-models-using-aws-himanshu-
singh/
textbookfull.com
Pramod Singh

Learn PySpark
Build Python-based Machine Learning and Deep
Learning Models
Pramod Singh
Bangalore, Karnataka, India

Any source code or other supplementary material referenced by the

author in this book is available to readers on GitHub via the book’s
product page, located at www.​apress.​com/​978-1-4842-4960-4 . For
more detailed information, please visit www.​apress.​com/​source-code .

ISBN 978-1-4842-4960-4 e-ISBN 978-1-4842-4961-1

https://doi.org/10.1007/978-1-4842-4961-1

© Pramod Singh 2019

Apress Standard

Trademarked names, logos, and images may appear in this book. Rather
than use a trademark symbol with every occurrence of a trademarked
name, logo, or image, we use the names, logos, and images only in an
editorial fashion and to the benefit of the trademark owner, with no
intention of infringement of the trademark. The use in this publication
of trade names, trademarks, service marks, and similar terms, even if
they are not identified as such, is not to be taken as an expression of
opinion as to whether or not they are subject to proprietary rights.

While the advice and information in this book are believed to be true
and accurate at the date of publication, neither the author nor the
editors nor the publisher can accept any legal responsibility for any
errors or omissions that may be made. The publisher makes no
warranty, express or implied, with respect to the material contained
herein.

Distributed to the book trade worldwide by Springer Science+Business

Media New York, 233 Spring Street, 6th Floor, New York, NY 10013.
Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-
[email protected], or visit www.springeronline.com. Apress Media,
LLC is a California LLC and the sole member (owner) is Springer
Science+Business Media Finance Inc (SSBM Finance Inc). SSBM Finance
Inc is a Delaware corporation.
I dedicate this book to my wife, Neha, my son, Ziaan, and my parents.
Without you, this book wouldn’t have been possible. You complete my
world and are the source of my strength.
Introduction
The idea of writing this book had already been seeded while I was
working on my first book, and there was a strong reason for that. The
earlier book was more focused on machine learning using big data and
essentially did not deep-dive sufficiently into supporting aspects, but
this book goes a little deeper into the internals of Spark’s machine
learning library, as well as analyzing of streaming data. It is a good
reference point for someone who wants to learn more about how to
automate different workflows and build pipelines to handle real-time
data.
This book is divided into three main sections. The first provides an
introduction to Spark and data analysis on big data; the middle section
discusses using Airflow for executing different jobs, in addition to data
analysis on streaming data, using the structured streaming features of
Spark. The final section covers translation of a business problem into
machine learning and solving it, using Spark’s machine learning library,
with a deep dive into deep learning as well.
This book might also be useful to data analysts and data engineers,
as it covers the steps of big data processing using PySpark. Readers
who want to make a transition to the data science and machine learning
fields will also find this book a good starting point and can gradually
tackle more complicated areas later. The case studies and examples
given in the book make it really easy to follow and understand the
related fundamental concepts. Moreover, there are very few books
available on PySpark, and this book certainly adds value to readers’
knowledge. The strength of this book lies in its simplicity and on its
application of machine learning to meaningful datasets.
I have tried my best to put all my experience and knowledge into
this book, and I feel it is particularly relevant to what businesses are
seeking in order to solve real challenges. I hope that it will provide you
with some useful takeaways.
Acknowledgments
This is my second book on Spark, and along the way, I have come to
realize my love for handling big data and performing machine learning
as well. Going forward, I intend to write many more books, but first, let
me thank a few people who have helped me along this journey. First, I
must thank the most important person in my life, my beloved wife,
Neha, who selflessly supported me throughout and sacrificed so much
to ensure that I completed this book.
I must thank Celestin Suresh John, who believed in me and extended
the opportunity to write this book. Aditee Mirashi is one of the best
editors in India. This is my second book with her, and it was even more
exciting to work with her this time. As usual, she was extremely
supportive and always there to accommodate my requests. I especially
would like to thank Jim Markham, who dedicated his time to reading
every single chapter and offered so many useful suggestions. Thanks,
Jim, I really appreciate your input. I also want to thank Manoj Patil, who
had the patience to review every line of code and check the
appropriateness of each example. Thank you for your feedback and
encouragement. It really made a difference to me and the book.
I also want to thank the many mentors who have constantly forced
me to pursue my dreams. Thank you Sebastian Keupers, Dr. Vijay
Agneeswaran, Sreenivas Venkatraman, Shoaib Ahmed, and Abhishek
Kumar, for your time. Finally, I am infinitely grateful to my son, Ziaan,
and my parents, for their endless love and support, irrespective of
circumstances. You all make my world beautiful.
Table of Contents
Chapter 1:​Introduction to Spark
History
Data Collection
Data Storage
Data Processing
Spark Architecture
Storage
Resource Management
Engine and Ecosystem
Programming Language APIs
Setting Up Your Environment
Local Setup
Dockers
Cloud Environments
Conclusion
Chapter 2:​Data Processing
Creating a SparkSession Object
Creating Dataframes
Null Values
Subset of a Dataframe
Select
Filter
Where
Aggregations
Collect
 User-Defined Functions (UDFs)
Pandas UDF
Joins
Pivoting
Window Functions or Windowed Aggregates
Conclusion
Chapter 3:​Spark Structured Streaming
Batch vs.​Stream
Batch Data
Stream Processing
Spark Streaming
Structured Streaming
Data Input
Data Processing
Final Output
Building a Structured App
Operations
Joins
Structured Streaming Alternatives
Conclusion
Chapter 4:​Airflow
Workflows
Graph Overview
Undirected Graphs
Directed Graphs
DAG Overview
 Operators
Installing Airflow
Airflow Using Docker
Creating Your First DAG
Step 1:​Importing the Required Libraries
Step 2:​Defining the Default Arguments
Step 3:​Creating a DAG
Step 4:​Declaring Tasks
Step 5:​Mentioning Dependencies
Conclusion
Chapter 5:​MLlib:​Machine Learning Library
Calculating Correlations
Chi-Square Test
Transformations
Binarizer
Principal Component Analysis
Normalizer
Standard Scaling
Min-Max Scaling
MaxAbsScaler
Binning
Building a Classification Model
Step 1:​Load the Dataset
Step 2:​Explore the Dataframe
Step 3:​Data Transformation
Step 4:​Splitting into Train and Test Data
 Step 5:​Model Training
Step 6:​Hyperparameter Tuning
Step 7:​Best Model
Conclusion
Chapter 6:​Supervised Machine Learning
Supervised Machine Learning Primer
Binary Classification
Multi-class Classification
Building a Linear Regression Model
Reviewing the Data Information
Generalized Linear Model Regression
Decision Tree Regression
Random Forest Regressors
Gradient-Boosted Tree Regressor
Step 1:​Build and Train a GBT Regressor Model
Step 2:​Evaluate the Model Performance on Test Data
Building Multiple Models for Binary Classification Tasks
Logistic Regression
Decision Tree Classifier
Support Vector Machines Classifiers
Naive Bayes Classifier
Gradient Boosted Tree Classifier
Random Forest Classifier
Hyperparameter Tuning and Cross-Validation
Conclusion
Chapter 7:​Unsupervised Machine Learning
 Unsupervised Machine Learning Primer
Reviewing the Dataset
Importing SparkSession and Creating an Object
Reshaping a Dataframe for Clustering
Building Clusters with K-Means
Conclusion
Chapter 8:​Deep Learning Using PySpark
Deep Learning Fundamentals
Human Brain Neuron vs.​Artificial Neuron
Activation Functions
Neuron Computation
Training Process:​Neural Network
Building a Multilayer Perceptron Model
Conclusion
Index
About the Author and About the Technical
Reviewer

About the Author

Pramod Singh
has more than 11 years of hands-on
experience in data engineering and
sciences and is currently a manager
(data science) at Publicis Sapient in
India, where he drives strategic
initiatives that deal with machine
learning and artificial intelligence (AI).
Pramod has worked with multiple
clients, in areas such as retail, telecom,
and automobile and consumer goods,
and is the author of Machine Learning
with PyS park . He also speaks at major
forums, such as Strata Data, and at AI
conferences.
Pramod received a bachelor’s degree in electrical and electronics
engineering from Mumbai University and an MBA (operations and
finance) from Symbiosis International University, in addition to data
analytics certification from IIM–Calcutta.
Pramod lives in Bangalore with his wife and three-year-old son. In
his spare time, he enjoys playing guitar, coding, reading, and watching
soccer.

About the Technical Reviewer

Manoj Patil
has worked in the software industry for 19 years. He received an
engineering degree from COEP, Pune (India), and has been enjoying his
exciting IT journey ever since.
 As a principal architect at TatvaSoft,
Manoj has taken many initiatives in the
organization, ranging from training and
mentoring teams, leading data science
and ML practice, to successfully
designing client solutions from different
functional domains.
He began his career as a Java
programmer but is fortunate to have
worked on multiple frameworks with
multiple languages and can claim to be a
full stack developer. In the last five years,
Manoj has worked extensively in the
field of BI, big data, and machine
learning, using such technologies as
Hitachi Vantara (Pentaho), the Hadoop ecosystem, TensorFlow, Python-
based libraries, and more.
He is passionate about learning new technologies, trends, and
reviewing books. When he’s not working, he’s either exercising or
reading/listening to infinitheism literature.
© Pramod Singh 2019
P. Singh, Learn PySpark
https://doi.org/10.1007/978-1-4842-4961-1_1

1. Introduction to Spark
Pramod Singh1

(1) Bangalore, Karnataka, India

As this book is about Spark, it makes perfect sense to start the first
chapter by looking into some of Spark’s history and its different
components. This introductory chapter is divided into three sections. In
the first, I go over the evolution of data and how it got as far as it has, in
terms of size. I’ll touch on three key aspects of data. In the second
section, I delve into the internals of Spark and go over the details of its
different components, including its architecture and modus operandi.
The third and final section of this chapter focuses on how to use Spark
in a cloud environment.

History
The birth of the Spark project occurred at the Algorithms, Machine, and
People (AMP) Lab at the University of California, Berkeley. The project
was initiated to address the potential issues in the Hadoop MapReduce
framework. Although Hadoop MapReduce was a groundbreaking
framework to handle big data processing, in reality, it still had a lot of
limitations in terms of speed. Spark was new and capable of doing in-
memory computations, which made it almost 100 times faster than any
other big data processing framework. Since then, there has been a
continuous increase in adoption of Spark across the globe for big data
applications. But before jumping into the specifics of Spark, let’s
consider a few aspects of data itself.
 Data can be viewed from three different angles: the way it is
collected, stored, and processed, as shown in Figure 1-1.

Figure 1-1 Three aspects of data

Data Collection
A huge shift in the manner in which data is collected has occurred over
the last few years. From buying an apple at a grocery store to deleting
an app on your mobile phone, every data point is now captured in the
back end and collected through various built-in applications. Different
Internet of things (IoT) devices capture a wide range of visual and
sensory signals every millisecond. It has become relatively convenient
for businesses to collect that data from various sources and use it later
for improved decision making.

Data Storage
In previous years, no one ever imagined that data would reside at some
remote location, or that the cost to store data would be as cheap as it is.
Businesses have embraced cloud storage and started to see its benefits
over on-premise approaches. However, some businesses still opt for on-
premise storage, for various reasons. It’s known that data storage
began by making use of magnetic tapes. Then the breakthrough
introduction of floppy discs made it possible to move data from one
place to another. However, the size of the data was still a huge
limitation. Flash drives and hard discs made it even easier to store and
transfer large amounts of data at a reduced cost. (See Figure 1-2.) The
latest trend in the advancement of storage devices has resulted in flash
drives capable of storing data up to 2TBs, at a throwaway price.

Figure 1-2 Evolution of data storage

This trend clearly indicates that the cost to store data has been
reduced significantly over the years and continues to decline. As a
result, businesses don’t shy away from storing huge amounts of data,
irrespective of its kind. From logs to financial and operational
transactions to simple employee feedback, everything gets stored.

Data Processing
The final aspect of data is using stored data and processing it for some
analysis or to run an application. We have witnessed how efficient
computers have become in the last 20 years. What used to take five
minutes to execute probably takes less than a second using today’s
machines with advanced processing units. Hence, it goes without saying
that machines can process data much faster and easier. Nonetheless,
there is still a limit to the amount of data a single machine can process,
regardless of its processing power. So, the underlying idea behind Spark
is to use a collection (cluster) of machines and a unified processing
engine (Spark) to process and handle huge amounts of data, without
compromising on speed and security. This was the ultimate goal that
resulted in the birth of Spark.

Spark Architecture
There are five core components that make Spark so powerful and easy
to use. The core architecture of Spark consists of the following layers, as
shown in Figure 1-3:
Storage
Resource management
Engine
Ecosystem
APIs

Figure 1-3 Core components of Spark

Storage
Before using Spark, data must be made available in order to process it.
This data can reside in any kind of database. Spark offers multiple
options to use different categories of data sources, to be able to process
it on a large scale. Spark allows you to use traditional relational
databases as well as NoSQL, such as Cassandra and MongoDB.

Resource Management
The next layer consists of a resource manager. As Spark works on a set
of machines (it also can work on a single machine with multiple cores),
it is known as a Spark cluster . Typically, there is a resource manager in
any cluster that efficiently handles the workload between these
resources. The two most widely used resource managers are YARN and
Mesos. The resource manager has two main components internally:
1. Cluster manager

2. Worker

It’s kind of like master-slave architecture, in which the cluster

manager acts as a master node, and the worker acts as a slave node in
the cluster. The cluster manager keeps track of all information
pertaining to the worker nodes and their current status. Cluster
managers always maintain the following information:
Status of worker node (busy/available)
Location of worker node
Memory of worker node
Total CPU cores of worker node
The main role of the cluster manager is to manage the worker nodes
and assign them tasks, based on the availability and capacity of the
worker node. On the other hand, a worker node is only responsible for
executing the task it’s given by the cluster manager, as shown in Figure
1-4.
Figure 1-4 Resource management
The tasks that are given to the worker nodes are generally the
individual pieces of the overall Spark application. The Spark application
contains two parts:
1. Task

2. Spark driver

The task is the data processing logic that has been written in either
PySpark or Spark R code. It can be as simple as taking a total frequency
count of words to a very complex set of instructions on an unstructured
dataset. The second component is Spark driver, the main controller of a
Spark application, which consistently interacts with a cluster manager
to find out which worker nodes can be used to execute the request. The
role of the Spark driver is to request the cluster manager to initiate the
Spark executor for every worker node.

Engine and Ecosystem

The base of the Spark architecture is its core, which is built on top of
RDDs (Resilient Distributed Datasets) and offers multiple APIs for
building other libraries and ecosystems by Spark contributors. It
contains two parts: the distributed computing infrastructure and the
RDD programming abstraction. The default libraries in the Spark toolkit
come as four different offerings.

Spark SQL
SQL being used by most of the ETL operators across the globe makes it
a logical choice to be part of Spark offerings. It allows Spark users to
perform structured data processing by running SQL queries. In
actuality, Spark SQL leverages the catalyst optimizer to perform the
optimizations during the execution of SQL queries.
Another advantage of using Spark SQL is that it can easily deal with
multiple database files and storage systems such as SQL, NoSQL,
Parquet, etc.

MLlib
Training machine learning models on big datasets was starting to
become a huge challenge, until Spark’s MLlib (Machine Learning
library) came into existence. MLlib gives you the ability to train
machine learning models on huge datasets, using Spark clusters. It
allows you to build in supervised, unsupervised, and recommender
systems; NLP-based models; and deep learning, as well as within the
Spark ML library.

Structured Streaming
The Spark Streaming library provides the functionality to read and
process real-time streaming data. The incoming data can be batch data
or near real-time data from different sources. Structured Streaming is
capable of ingesting real-time data from such sources as Flume, Kafka,
Twitter, etc. There is a dedicated chapter on this component later in this
book (see Chapter 3).

Graph X
This is a library that sits on top of the Spark core and allows users to
process specific types of data (graph dataframes), which consists of
nodes and edges. A typical graph is used to model the relationship
between the different objects involved. The nodes represent the object,
and the edge between the nodes represents the relationship between
them. Graph dataframes are mainly used in network analysis, and
Graph X makes it possible to have distributed processing of such graph
dataframes.

Programming Language APIs

Spark is available in four languages. Because Spark is built using Scala,
that becomes the native language. Apart from Scala, we can also use
Python, Java, and R, as shown in Figure 1-5.

Figure 1-5 Language APIs

Setting Up Your Environment

In this final section of this chapter, I will go over how to set up the
Spark environment in the cloud. There are multiple ways in which we
can use Spark:
 Local setup
Dockers
Cloud environment (GCP, AWS, Azure)
Databricks

Local Setup
It is relatively easy to install and use Spark on a local system, but it fails
the core purpose of Spark itself, if it’s not used on a cluster. Spark’s core
offering is distributed data processing, which will always be limited to a
local system’s capacity, in the case that it’s run on a local system,
whereas one can benefit more by using Spark on a group of machines
instead. However, it is always good practice to have Spark locally, as
well as to test code on sample data. So, follow these steps to do so:
1. Ensure that Java is installed; otherwise install Java.

2. Download the latest version of Apache Spark from

https://spark.apache.org/downloads.html .

3. Extract the files from the zipped folder.

4. Copy all the Spark-related files to their respective directory.

5. Configure the environment variables to be able to run Spark.

6. Verify the installation and run Spark.

Dockers
Another way of using Spark locally is through the containerization
technique of dockers . This allows users to wrap all the dependencies
and Spark files into a single image, which can be run on any system. We
can kill the container after the task is finished and rerun it, if required.
To use dockers for running Spark, we must install Docker on the system
first and then simply run the following command: [In]: docker
run -it -p 8888:8888 jupyter/pyspark-notebook".
Cloud Environments
As discussed earlier in this chapter, for various reasons, local sets are
not of much help when it comes to big data, and that’s where cloud-
based environments make it possible to ingest and process huge
datasets in a short period. The real power of Spark can be seen easily
while dealing with large datasets (in excess of 100TB). Most of the
cloud-based infra-providers allow you to install Spark, which
sometimes comes preconfigured as well. One can easily spin up the
clusters with required specifications, according to need. One of the
cloud-based environments is Databricks.

Databricks
Databricks is a company founded by the creators of Spark, in order to
provide the enterprise version of Spark to businesses, in addition to
full-fledged support. To increase Spark’s adoption among the
community and other users, Databricks also provides a free community
edition of Spark, with a 6GB cluster (single node). You can increase the
size of the cluster by signing up for an enterprise account with
Databricks, using the following steps:
1. Search for the Databricks web site and select Databricks
Community Edition, as shown in Figure 1-6.

Figure 1-6 Databricks web page

2. If you have a user account with Databricks, you can simply log in. If
you don’t have an account, you must create one, in order to use
Databricks, as shown in Figure 1-7.

Figure 1-7 Databricks login

3. Once you are on the home page, you can choose to either load a
new data source or create a notebook from scratch, as shown in
Figure 1-8. In the latter case, you must have the cluster up and
running, to be able to use the notebook. Therefore, you must click
New Cluster, to spin up the cluster. (Databricks provides a 6GB AWS
EMR cluster.)
 Figure 1-8 Creating a Databricks notebook
4. To set up the cluster, you must give a name to the cluster and select
the version of Spark that must configure with the Python version,
as shown in Figure 1-9. Once all the details are filled in, you must
click Create Cluster and wait a couple of minutes, until it spins up.

Figure 1-9 Creating a Databricks cluster

5. You can also view the status of the cluster by going into the Clusters
option on the left side widget, as shown in Figure 1-10. It gives all
 the information associated with the particular cluster and its
current status.

Figure 1-10 Databricks cluster list

6. The final step is to open a notebook and attach it to the cluster you
just created (Figure 1-11). Once attached, you can start the PySpark
code.

Figure 1-11 Databricks notebook

Overall, since 2010, when Spark became an open source platform,

its users have risen in number consistently, and the community
continues to grow every day. It’s no surprise that the number of
contributors to Spark has outpaced that of Hadoop. Some of the reasons
for Spark’s popularity were noted in a survey, the results of which are
shown in Figure 1-12.
Figure 1-12 Results of Spark adoption survey

Conclusion
This chapter provided a brief history of Spark, its core components, and
the process of accessing it in a cloud environment. In upcoming
chapters, I will delve deeper into the various aspects of Spark and how
to build different applications with it.
© Pramod Singh 2019
P. Singh, Learn PySpark
https://doi.org/10.1007/978-1-4842-4961-1_2

2. Data Processing
Pramod Singh1

(1) Bangalore, Karnataka, India

This chapter covers different steps to preprocess and handle data in

PySpark. Preprocessing techniques can certainly vary from case to case,
and many different methods can be used to massage the data into
desired form. The idea of this chapter is to expose some of the common
techniques for dealing with big data in Spark. In this chapter, we are
going to go over different steps involved in preprocessing data, such as
handling missing values, merging datasets, applying functions,
aggregations, and sorting. One major part of data preprocessing is the
transformation of numerical columns into categorical ones and vice
versa, which we are going to look at over the next few chapters and are
based on machine learning. The dataset that we are going to make use
of in this chapter is inspired by a primary research dataset and contains
a few attributes from the original dataset, with additional columns
containing fabricated data points.

Note All the following steps are written in Jupyter Notebook,

running Spark on a Docker image (mentioned in Chapter 1). All the
subsequent code can also be run in Databricks.

Creating a SparkSession Object

The first step is to create a SparkSession object , in order to use
Spark. We also import all the required functions and datatypes from
 Another Random Document on
Scribd Without Any Related Topics
 “THE GULLET SCIENCE”
A LOOK BACK AND AN ECONOMIC FORECAST

The cook-book is not a modern product. The Iliad is the hungriest

book on earth, and it is the first of our cook-books aside from half-
sacred, half-sanitary directions to the early Aryans and Jews. It is
that acme of poetry, that most picturesque of pictures, that most
historical of histories, that most musical and delicious verse, the
Iliad, which was the first popularly to teach the cooking art—the art
in its simplicity, and not a mere handmaid to sanitation,
jurisprudence, or theology. Through the pages of that great poem
blow not only the salt winds of the Ægean Sea, but also the savor of
tender kid and succulent pig, not to mention whole hectacombs,
which delighted the blessed gods above and strengthened hungry
heroes below. To this very day—its realism is so perfect—we catch
the scent of the cooking and see the appetiteful people eat. The
book is half-human, half-divine; and in its human part the pleasures
and the economic values of wholesome fare are not left out.

No, cook-books are not modern products. They were in Greece later
than Homer. When the Greek states came to the fore in their
wonderful art and literature and the distinction of a free democracy,
plain living characterized nearly all the peoples. The Athenians were
noted for their simple diet. The Spartans were temperate to a
proverb, and their συσσίτια (public meals), later called φειδίτια
(spare meals), guarded against indulgence in eating. To be a good
cook was to be banished from Sparta.

But with the Western Greeks, the Greeks of Sicily and Southern Italy,
it was different—those people who left behind them little record of
the spirit. In Sybaris the cook who distinguished himself in preparing
a public feast—such festivals being not uncommon—received a
crown of gold and the freedom of the games. It was a citizen of that
luxury-loving town who averred, when he tasted the famous black
soup, that it was no longer a wonder the Spartans were fearless in
battle, for any one would readily die rather than live on such a diet.
Among the later Greeks the best cooks, and the best-paid cooks,
came from Sicily; and that little island grew in fame for its gluttons.

There is a Greek book—the Deipnosophistæ—Supper of the “Wise

Men—written by Athenæus—which holds for us much information
about the food and feasting of those old Hellenes. The wise men at
their supposed banquet quote, touching food and cooking, from
countless Greek authors whose works are now lost, but were still
preserved in the time of Athenæus. This, for instance, is from a
poem by Philoxenus of Cythera, who wittily and gluttonously lived at
the court of Dionysius of Syracuse, and wished for a throat three
cubits long that the delight of tasting might be drawn out.3
 “And then two slaves brought in a well-rubb’d table.
.... Then came a platter
.... with dainty sword-fish fraught,
And then fat cuttle-fish, and the savoury tribes
Of the long hairy polypus. After this
Another orb appear’d upon the table,
Rival of that just brought from off the fire,
Fragrant with spicy odour. And on that
Again were famous cuttle-fish, and those
Fair maids the honey’d squills, and dainty cakes,
Sweet to the palate, and large buns of wheat,
Large as a partridge, sweet and round, which you
Do know the taste of well. And if you ask
What more was there, I’d speak of luscious chine,
And loin of pork, and head of boar, all hot;
Cutlets of kid, and well-boil’d pettitoes,
And ribs of beef, and heads, and snouts and tails,
Then kid again, and lamb, and hares, and poultry,
Partridges and the bird from Phasis’ stream.
And golden honey, and clotted cream was there,
And cheese which I did join with all in calling
Most tender fare.”

The Greeks used many of the meats and vegetables we enjoy; and
others we disclaim; for instance, cranes. Even mushrooms were
known to their cooks, and Athenæus suggests how the wholesome
may be distinguished from the poisonous, and what antidotes serve
best in case the bad are eaten. But with further directions of his our
tastes would not agree. He recommends seasoning the mushrooms
with vinegar, or honey and vinegar, or honey, or salt—for by these
means their choking properties are taken away.

The writings of Athenæus have, however, a certain literary and, for

his time as well as our own, an historic and archæologic flavor. The
only ancient cook-book pure and simple—bent on instruction in the
excellent art—which has come down to us is that of Apicius, in ten
short books, or chapters. And which Apicius? Probably the second of
the name, the one who lectured on cooking in Rome during the
reign of Augustus. He gave some very simple directions which hold
good to the present day; for instance—

“UT CARNEM SALSAM DULCEM FACIAS

“Carnem salsam dulcem facies, si prius in lacte coquas, et postea in

aqua.”

But again his compounds are nauseating even in print. He was

famous for many dishes, and Pliny, in his Natural History, says he
discovered the way of increasing the size of the liver of the pig—just
as the liver of the Strasbourg geese is enlarged for pâté de foie gras,
and as our own Southern people used to induce pathological
conditions in their turkeys.

The method of Apicius was to cram the pig with dried figs, and,
when it was fat enough, drench it with wine mixed with honey.
“There is,” continues Pliny, “no other animal that affords so great a
variety to the palate; all others have their taste, but the pig fifty
different flavors. From this tastiness of the meat it came about that
the censors made whole pages of regulations about serving at
banquets the belly and the jowls and other dainty parts. But in spite
of their rules the poet Publius, author of the Mimes, when he ceased
to be a slave, is said never to have given an entertainment without a
dish of pig’s belly which he called ‘sumen.’”

“Cook Apicius showed a remarkable ingenuity in developing luxury,”

the old Roman says at another time, “and thought it a most
excellent plan to let a mullet die in the pickle known as ‘garum.’” It
was ingenuity of cruelty as well as of luxury. “They killed the fish in
sauces and pickled them alive at the banquet,” says Seneca,
“feeding the eye before the gullet, for they took pleasure in seeing
their mullets change several colors while dying.” The unthinkable
garum was made, according to Pliny, from the intestines of fish
macerated with salt, and other ingredients were added before the
mixture was set in the sun to putrefy and came to the right point for
serving. It also had popularity as a household remedy for dog-bites,
etc.; and in burns, when care was necessary in its application not to
mention it by name—so delicately timid was its healing spirit. Its use
as a dish was widespread, and perhaps we see in the well-known
hankerings of the royal George of England a reversion to the palate
of Italian ancestors.

But garum was only one of strange dishes. The Romans seasoned
much with rue and asafetida!—a taste kept to this day in India,
where “Kim” eats “good curry cakes all warm and well-scented with
hing (asafetida).” Cabbages they highly estimated; “of all garden
vegetables they thought them best,” says Pliny. The same author
notes that Apicius rejected Brussels sprouts, and in this was followed
by Drusus Cæsar, who was censured for over-nicety by his father,
the Emperor Tiberius of Capreæ villas fame.

Upon cooks and the Roman estimate of their value in his day Pliny
also casts light. “Asinius Celer, a man of consular rank and noted for
his expenditure on mullet, bought one at Rome during the reign of
Gaius Caligula for eight thousand sesterces. Reflection on this fact,”
continues Pliny, “will recall the complaints uttered against luxury and
the lament that a single cook costs more than a horse. At the
present day a cook is only to be had for the price of a triumph, and
a mullet only to be had for what was once the price of a cook! Of a
fact there is now hardly any living being held in higher esteem than
the man who knows how to get rid of his master’s belongings in the
most scientific fashion!”

Much has been written of the luxury and enervation of Romans after
the republic, how they feasted scented with perfumes, reclining and
listening to music, “nudis puellis ministrantibus.” The story is old of
how Vedius Pollio “hung with ecstasy over lampreys fattened on
human flesh;” how Tiberius spent two days and two nights in one
bout; how Claudius dissolved pearls for his food; how Vitellius
delighted in the brains of pheasants and tongues of nightingales and
the roe of fish difficult to take; how the favorite supper of
Heliogabalus was the brains of six hundred thrushes. At the time
these gluttonies went on in the houses of government officials, the
mass of the people, the great workers who supported the great
idlers, fed healthfully on a mess of pottage. The many to support the
super-abundant luxury of a few is still one of the mysteries of the
people.

But in the old Rome the law of right and honest strength at last
prevailed, and monsters gave way to the cleaner and hardier chiefs
of the north. The mastery of the world necessarily passed to others;
—it has never lain with slaves of the stomach.

The early folk of Britain—those Cæesar found in the land from which
we sprang—ate the milk and flesh of their flocks. They made bread
by picking the grains from the ear and pounding them to paste in a
mortar. Their Roman conquerors doubtless brought to their midst a
more elaborated table order. Barbarous Saxons, fighters and
freebooters, next settling on the rich island and restraining
themselves little for sowing and reaping, must in their incursions
have been flesh-eaters, expeditiously roasting and broiling directly
over coals like our early pioneers.

This mode of living also would seem true of the later-coming Danes,
who after their settlement introduced, says Holinshed, another habit.
“The Danes,” says that delightful chronicler, “had their dwelling ...
among the Englishmen, whereby came great harme; for whereas the
Danes by nature were great drinkers, the Englishmen by continuall
conversation with them learned the same vice. King Edgar, to
reforme in part such excessive quaffing as then began to grow in
use, caused by the procurement of Dunstane [the then Archbishop
of Canterbury] nailes to be set in cups of a certeine measure,
marked for the purpose, that none should drinke more than was
assigned by such measured cups. Englishmen also learned of the
Saxons, Flemings, and other strangers, their peculiar kinds of vices,
as of the Saxons a disordered fierceness of mind, of the Flemings a
feeble tendernesse of bodie; where before they rejoiced in their
owne simplicitie and esteemed not the lewd and unprofitable
manners of strangers.”

But refinement was growing in the mixture of races which was to

make modern Englishmen, and in the time of Hardicanute, much
given to the pleasures of the table and at last dying from too
copious a draught of wine,—“he fell downe suddenlie,” says
Holinshed, “with the pot in his hand”—there was aim at niceness
and variety and hospitable cheer.

The Black Book of a royal household which Warner quotes in his

“Antiquitates Culinariæ”4 is evidence of this:

“Domus Eegis Hardeknoute may be called a fader noreshoure of

familiaritie, which used for his own table, never to be served with
ony like metes of one meale in another, and that chaunge and
diversitie was dayly in greate habundance, and that same after to be
ministred to his alms-dishe, he caused cunyng cooks in curiositie;
also, he was the furst that began four meales stablyshed in oon day,
opynly to be holden for worshuppfull and honest peopull resorting to
his courte; and no more melis, nor brekefast, nor chambyr, but for
his children in householde; for which four melys he ordeyned four
marshalls, to kepe the honor of his halle in recevyng and dyrecting
strangers, as well as of his householdemen in theyre fitting, and for
services and ther precepts to be obeyd in. And for the halle, with all
diligence of officers thereto assigned from his furst inception, tyll the
day of his dethe, his house stode after one unyformitie.”

Of Hardicanute, “it hath,” says Holinshed, “beene commonlie told,

that Englishmen learned of him their excessive gourmandizing and
unmeasurable filling of their panches with meates and drinkes,
whereby they forgat the vertuous use of sobrietie, so much
necessarie to all estates and degrees, so profitable for all
commonwealthes, and so commendable both in the sight of God,
and all good men.”

Not only to the Danes, but also to the later conquerors, the
Normans, the old chronicler attributes corruption of early English
frugality and simplicity. “The Normans, misliking the gormandise of
Canutus, ordeined after their arrivall that no table should be covered
above once in the day.... But in the end, either waxing wearie of
their owne frugalitie or suffering the cockle of old custome to
overgrow the good corne of their new constitution, they fell to such
libertie that in often feeding they surmounted Canutus surnamed the
hardie.... They brought in also the custome of long and statelie
sitting at meat.”

A fellow-Londoner with Holinshed, John Stow, says of the reign of

William Rufus, the second Norman king of England, “The courtiers
devoured the substance of the husbandmen, their tenants.”

And Stow’s “Annales” still further tell of a banquet served in far-off

Italy to the duke of Clarence, son of Edward III., when, some three
hundred years after the Norman settlement, the lad Leonell went to
marry Violentis, daughter of the duke of Milan. It should not be
forgotten that in the reign of Edward II. of England, grandfather of
the duke, proclamation had been issued against the “outrageous and
excessive multitude of meats and dishes” served by the nobles in
their castles, as well by “persons of inferior rank imitating their
example, beyond what their station required and their circumstances
could afford.”

“At the comming of Leonell”, says Stow, “such aboundance of

treasure was in most bounteous maner spent, in making most
sumptuous feasts, setting forth stately fightes, and honouring with
rare gifts above two hundred Englishmen, which accompanied his
[the duke of Milan’s] son-in-law, as it seemed to surpasse the
greatnesse of most wealthy Princes; for in the banquet whereat
Francis Petrarch was present, amongst the chiefest guestes, there
were above thirtie courses of service at the table, and betwixt every
course, as many presents of wonderous price intermixed, all which
John Galeasius, chiefe of the choice youth, bringing to the table, did
offer to Leonell ... And such was the sumptuousnesse of that
banquet, that the meats which were brought from the table, would
sufficiently have served ten thousand men.”

The first cook-book we have in our ample English tongue is of date

about 1390. Its forme, says the preface to the table of contents, this
“forme of cury [cookery] was compiled of the chef maistes cokes of
kyng Richard the Secunde kyng of nglond aftir the conquest; the
which was accounted the best and ryallest vyand [nice eater] of alle
csten ynges [Christian kings]; and it was compiled by assent and
avysement of maisters and [of] phisik and of philosophie that
dwellid in his court. First it techith a man for to make commune
pottages and commune meetis for howshold, as they shold be made,
craftly and holsomly. Aftirward it techith for to make curious
potages, and meetes, and sotiltees, for alle maner of states, bothe
hye and lowe. And the techyng of the forme of making of potages,
and of meetes, bothe of flesh, and of fissh, buth [are] y sette here
by noumbre and by ordre. Sso this little table here fewyng
[following] wole teche a man with oute taryyng, to fynde what
meete that hym lust for to have.”

The “potages” and “meetis” and “sotiltees” it techith a man for to

make would be hardly more endurable to the modern stomach than
some old Greek and Roman seasonings we have referred to. There is
no essential difference between these and the directions of a rival
cook-book written some forty or fifty years later and divided into
three parts—Kalendare de Potages dyvers, Kalendare de Leche
Metys, Dyverse bake metis. Or of another compiled about 1450. Let
us see how they would make a meat.

“Stwed Beeff. Take faire Ribbes of ffresh beef, And (if thou wilt)
roste hit til hit be nygh ynowe; then put hit in a faire possenet; caste
therto parcely and oynons mynced, reysons of corauns, powder
peper, canel, clowes, saundres, safferon, and salt; then caste thereto
wyn and a litull vynegre; sette a lyd on the potte, and lete hit boile
sokingly on a faire charcole til hit be ynogh; then lay the fflessh, in
disshes, and the sirippe thereuppon, And serve it forth.”

And for sweet apple fritters:

“Freetours. Take yolkes of egges, drawe hem thorgh a streynour,

caste thereto faire floure, berme and ale; stere it togidre till hit be
thik. Take pared appelles, cut hem thyn like obleies [wafers of the
eucharist], ley hem in the batur; then put hem into a ffrying pan,
and fry hem in faire grece or buttur til thei ben browne yelowe; then
put hem in disshes; and strawe Sugur on hem ynogh, And serve
hem forthe.”

Still other cook-books followed—the men of that day served hem

forthe—among which we notice “A noble Boke off Cookry ffor a
prynce houssolde or eny other estately houssolde,” ascribed to about
the year 1465.

To the monasteries the art of cooking is doubtless much indebted,

just as even at the present day is the art of making liqueurs. Their
vast wealth, the leisure of the in-dwellers, and the gross sensualism
and materialism of the time they were at their height would naturally
lead to care for the table and its viands. Within their thick stone
walls, which the religious devotion of the populace had reared, the
master of the kitchen, magister coquinæ or magnus coquus, was not
the man of least importance. Some old author whose name and
book do not come promptly to memory refers to the disinclination of
plump capons, or round-breasted duck, to meet ecclesiastical eyes—
a facetiousness repeated in our day when the Uncle Remuses of
Dixie say they see yellow-legged chickens run and hide if a preacher
drives up to supper.
Moreover, the monasteries were the inns of that day where travellers
put up, and in many instances were served free—no price, that is,
was put upon their entertainment, the abbot, or the establishment,
receiving whatever gift the one sheltered and fed felt able or moved
to pay.

Contemporary accounts of, or references to, the cooking and

feasting in religious houses are many—those of the Vision of Long
Will concerning Piers the Plowman, those of “Dan Chaucer, the first
warbler,” of Alexander Barclay, and Skelton, great satirist of times of
Henry VIII., and of other authors not so well remembered. Now and
then a racy anecdote has come down like that which Thomas Fuller
saves from lip tradition in his “History of Abbeys in England.” It
happened, says Worthy Fuller, that Harry VIII., “hunting in Windsor
Forest, either casually lost, or (more probable) wilfully losing himself,
struck down about dinner-time to the abbey of Reading; where,
disguising himself (much for delight, more for discovery, to see
unseen), he was invited to the abbot’s table, and passed for one of
the king’s guard, a place to which the proportion of his person might
properly entitle him. A sirloin of beef was set before him (so
knighted saith tradition, by this King Henry), on which the king laid
on lustily, not disgracing one of that place for whom he was
mistaken.

“‘Well fare thy heart!’ quoth the abbot; ‘and here in a cup of sack I
remember the health of his grace your master. I would give an
hundred pounds on the condition I could feed so heartily on beef as
you do. Alas! my weak and squeazy stomach will badly digest the
wing of a small rabbit or chicken.’

“The king pleasantly pledged him, and, heartily thanking him for his
good cheer, after dinner departed as undiscovered as he came
thither.

“Some weeks after, the abbot was sent for by a pursuivant, brought
up to London, clapped in the Tower, kept close prisoner, fed for a
short time with bread and water; yet not so empty his body of food,
as his mind was filled with fears, creating many suspicions to himself
when and how he had incurred the king’s displeasure. At last a
sirloin of beef was set before him, on which the abbot fed as the
farmer of his grange, and verified the proverb, that ‘Two hungry
meals make the third a glutton.’

“In springs King Henry out of a private lobby, where he had placed
himself, the invisible spectator of the abbot’s behavior. ‘My lord,’
quoth the king, ‘presently deposit your hundred pounds in gold, or
else no going hence all the days of your life. I have been your
physician to cure you of your squeazy stomach; and here, as I
deserve, I demand my fee for the same!’

“The abbot down with his dust; and, glad he had escaped so,
returned to Reading, as somewhat lighter in purse, so much more
merrier in heart than when he came thence.”

The “squeazy” abbot stood alone in proclamation of his disorder.

Archbishop Cranmer, according to John Leland, king’s antiquary to
Henry VIII., found it necessary in 1541 to regulate the expenses of
the tables of bishops and clergy by a constitution—an instrument
which throws much light on the then conditions, and which ran as
follows:

“In the yeare of our Lord MDXLI it was agreed and condescended
upon, as wel by the common consent of both tharchbishops and
most part of the bishops within this realme of Englande, as also of
divers grave men at that tyme, both deanes and archdeacons, the
fare at their tables to be thus moderated.

“First, that tharchbishop should never exceede six divers kindes of

fleshe, or six of fishe, on the fishe days; the bishop not to exceede
five, the deane and archdeacon not above four, and al other under
that degree not above three; provided also that tharchbishop myght
have of second dishes four, the bishop three, and al others under the
degree of a bishop but two. As custard, tart, fritter, cheese or
apples, peares, or two of other kindes of fruites. Provided also, that
if any of the inferior degree dyd receave at their table, any
archbishop, bishop, deane, or archdeacon, or any of the laitie of lyke
degree, viz. duke, marques, earle, viscount, baron, lorde, knyght,
they myght have such provision as were mete and requisite for their
degrees. Provided alway that no rate was limited in the receavying
of any ambassadour. It was also provided that of the greater fyshes
or fowles, there should be but one in a dishe, as crane, swan, turkey
cocke, hadocke, pyke, tench; and of lesse sortes but two, viz.
capons two, pheasantes two, conies two, and woodcockes two. Of
lesse sortes, as of patriches, the archbishop three, the bishop and
other degrees under hym two. Of blackburdes, the archbishop six,
the bishop four, the other degrees three. Of larkes and snytes
(snipes) and of that sort but twelve. It was also provided, that
whatsoever is spared by the cutting of, of the olde superfluitie,
shoulde yet be provided and spent in playne meates for the
relievyng of the poore. Memorandum, that this order was kept for
two or three monethes, tyll by the disusyng of certaine wylful
persons it came to the olde excesse.”

Still one more tale bearing upon a member of the clergy who would
set out more “blackburdes” than “tharchbishop” is told by Holinshed.
It has within it somewhat of the flavor of the odium theologicum,
but an added interest also, since it turns upon a dish esteemed in
Italy since the time of the imperial Romans—peacock, often served
even nowadays encased in its most wonderful plumage. The Pope
Julius III., whose luxurious entertainment and comport shocked the
proprieties even of that day, and who died in Rome while the
chronicler was busy in London, is the chief actor.

“At an other time,” writes Holinshed, “he sitting at dinner, pointing to

a peacocke upon his table, which he had not touched; Keepe (said
he) this cold peacocke for me against supper, and let me sup in the
garden, for I shall have ghests. So when supper came, and amongst
other hot peacockes, he saw not his cold peacocke brought to his
table; the pope after his wonted manner, most horriblie blaspheming
God, fell into an extreame rage, &c. Whereupon one of his cardinals
sitting by, desired him saieng: Let not your holinesse, I praie you, be
so mooved with a matter of so small weight. Then this Julius the
pope answeringe againe: What (saith he) if God was so angrie for
one apple, that he cast our first parents out of paradise for the
same, whie maie not I being his vicar, be angrie then for a peacocke,
sithens a peacocke is a greater matter than an apple.”

In England at this time controlling the laity were sumptuary laws,

habits of living resulting from those laws, and great inequalities in
the distribution of wealth. On these points Holinshed again brings us
light:

“In number of dishes and change of meat,” he writes, “the nobilitie

of England (whose cookes are for the most part musicall-headed
Frenchmen and strangers) do most exceed, sith there is no daie in
maner that passeth over their heads, wherein they have not onelie
beefe, mutton, veale, lambe, kid, porke, conie, capon, pig, or so
manie of these as the season yeeldeth; but also some portion of the
red or fallow deere, beside great varietie of fish and wild foule, and
thereto sundrie other delicates wherein the sweet hand of the
seasoning Portingale is not wanting; so that for a man to dine with
one of them, and to taste of everie dish that standeth before him ...
is rather to yeeld unto a conspiracie with a great deale of meat for
the speedie suppression of naturall health, then the use of a
necessarie meane to satisfie himselfe with a competent repast, to
susteine his bodie withall. But as this large feeding is not seene in
their gests, no more is it in their owne persons, for sith they have
dailie much resort unto their tables ... and thereto reteine great
numbers of servants, it is verie requisit and expedient for them to be
somewhat plentifull in this behalfe.

“The chiefe part likewise of their dailie provision is brought before

them ... and placed on their tables, whereof when they have taken
what it pleaseth them, the rest is reserved and afterwards sent
downe to their serving men and waiters, who feed thereon in like
sort with convenient moderation, their reversion also being
bestowed upon the poore, which lie readie at their gates in great
numbers to receive the same.

“The gentlemen and merchants keepe much about one rate, and
each of them contenteth himselfe with foure, five or six dishes,
when they have but small resort, or peradventure with one, or two,
or three at the most, when they have no strangers to accompanie
them at their tables. And yet their servants have their ordinarie diet
assigned, beside such as is left at their masters’ boordes, and not
appointed to be brought thither the second time, which
neverthelesse is often seene generallie in venison, lambe, or some
especiall dish, whereon the merchant man himselfe liketh to feed
when it is cold.”

“At such times as the merchants doo make their ordinarie or

voluntarie feasts, it is a world to see what great provision is made of
all maner of delicat meats, from everie quarter of the countrie....
They will seldome regard anie thing that the butcher usuallie killeth,
but reject the same as not worthie to come in place. In such cases
all gelisses of all coleurs mixed with a varitie in the representation of
sundrie floures, herbs, trees, formes of beasts, fish, foules and fruits,
and there unto marchpaine wrought with no small curiositie, tarts of
diverse hewes and sundrie denominations, conserves of old fruits
foren and homebred, suckets, codinacs, marmilats, marchpaine,
sugerbread, gingerbread, florentines, wild foule, venison of all sorts,
and sundrie outlandish confections altogither seasoned with sugar ...
doo generalie beare the swaie, beside infinit devises of our owne not
possible for me to remember. Of the potato and such venerous roots
as are brought out of Spaine, Portingale, and the Indies to furnish
our bankets, I speake not.”

“The artificer and husbandman make greatest accompt of such meat

as they may soonest come by, and have it quickliest readie.... Their
food also consisteth principallie in beefe and such meat as the
butcher selleth, that is to saie, mutton, veale, lambe, porke, etc., ...
beside souse, brawne, bacon, fruit, pies of fruit, foules of sundrie
sorts, cheese, butter, eggs, etc.... To conclude, both the artificer and
the husbandman are sufficientlie liberall and verie friendlie at their
tables, and when they meet they are so merie without malice and
plaine, without inward Italian or French craft and subtiltie, that it
would doo a man good to be in companie among them.

“With us the nobilitie, gentrie and students doo ordinarilie go to

dinner at eleven before noone, and to supper at five, or betweene
five and six at after-noone. The merchants dine and sup seldome
before twelve at noone, and six at night, especiallie in London. The
husbandmen dine also at high noone as they call it, and sup at
seven or eight.... As for the poorest sort they generallie dine and sup
when they may, so that to talke of their order of repast it were but a
needlesse matter.”

“The bread through out the land,” continues Holinshed, “is made of
such graine as the soil yeeldeth, neverthelesse the gentilitie
commonlie provide themselves sufficientlie of wheat for their owne
tables, whilst their houshold and poore neighbours in some shires
are inforced to content themselves with rie, or baricie, yea and in
time of dearth manie with bread made either of beans, or peason, or
otes, or of altogether and some acornes among.... There be much
more ground eared now almost in everie place than hath beene of
late yeares, yet such a price of come continueth in each towne and
market without any just cause (except it be that landlords doo get
licenses to carie come out of the land onelie to keepe up the prices
for their owne private games and ruine of the commonwealth), that
the artificer and poore laboring man is not able to reach unto it, but
is driven to content himselfe with horsse corne—I mean beanes,
peason, otes, tarres, and lintels.”
Books had been written for women and their tasks within—the
“Babees Booke,” Tusser’s5 “Hundrethe Good Pointes of Huswifry,”
“The Good Husive’s Handmaid”—the last two in the sixteenth
century; these and others of their kidney. A woman who thought,
spoke, and wrote in several tongues was greatly filling the throne of
England in those later times.

Cook- and receipt-books in the following century, that is in the

seventeenth, continued to discover women, and to realize moreover
that to them division of labor had delegated the household and its
businesses. There were “Jewels” and “Closets of Delights” before we
find an odd little volume putting out in 1655 a second edition. It
shows upon its title-page the survival from earlier conditions of the
confusion of duties of physician and cook—a fact made apparent in
the preface copied in the foregoing “forme of cury” of King Richard—
and perhaps intimates the housewife should perform the services of
both. It makes, as well, a distinct appeal to women as readers and
users of books. Again it evidences the growth of the Commons. In
full it introduces itself in this wise:

“The Ladies Cabinet enlarged and opened: containing Many Rare

Secrets and Rich Ornaments, of several kindes, and different uses.
Comprized under three general Heads, viz. of 1 Preserving,
Conserving, Candying, etc. 2 Physick and Chirurgery. 3 Cooking and
Housewifery. Whereunto is added Sundry Experiments and choice
Extractions of Waters, Oyls, etc. Collected and practised by the late
Right Honorable and Learned Chymist, the Lord Ruthuen.”

The preface, after an inscription “To the Industrious improvers of

Nature by Art; especially the vertuous Ladies and Gentlewomen of
the Land,” begins:

“Courteous Ladies, etc. The first Edition of this—(cal it what you

please) having received a kind entertainment from your Ladiships
hands, for reasons best known to yourselves, notwithstanding the
disorderly and confused jumbling together of things of different
kinds, hath made me (who am not a little concerned therein) to
bethink myself of some way, how to encourage and requite your
Ladiships Pains and Patience (vertues, indeed, of absolute necessity
in such brave employments; there being nothing excellent that is not
withal difficult) in the profitable spending of your vacant minutes.”
This labored and high-flying mode of address continues to the
preface’s end.... “I shall thus leave you at liberty as Lovers in
Gardens, to follow your own fancies. Take what you like, and delight
in your choice, and leave what you list to him, whose labour is not
lost if anything please.”

In turning the leaves of the book one comes upon such naïve
discourse as this:

“To make the face white and fair.

“Wash thy face with Rosemary boiled in white wine, and thou shalt
be fair; then take Erigan and stamp it, and take the juyce thereof,
and put it all together and wash thy face therewith. Proved.”

It was undoubtedly the success of “The Ladies Cabinet” and its

cousins german that led to the publication of a fourth edition in 1658
of another compilation, which, according to the preface, was to go
“like the good Samaritane giving comfort to all it met.” The title was
“The Queens Closet opened: Incomparable Secrets in Physick,
Chyrurgery, Preserving, Candying, and Cookery, As they were
presented unto the Queen By the most Experienced Persons of our
times.... Transcribed from the true Copies of her Majesties own
Receipt Books, by W. M. one of her late Servants.” It is curious to
recall that this book was published during the Cromwell Protectorate
—1658 is the year of the death of Oliver—and that the queen
alluded to in the title—whose portrait, engraved by the elder William
Faithorne, forms the frontispiece—was Henrietta Maria, widow of
Charles I., and at that time an exile in France.
During this century, which saw such publications as Rose’s “School
for the Officers of the Mouth,” and “Nature Unembowelled,” a
woman, Hannah Wolley, appears as author of “The Cook’s Guide.” All
such compilations have enduring human value, but we actually gain
quite as much of this oldest of arts from such records as those the
indefatigable Pepys left in his Diary. At that time men of our race did
not disdain a knowledge of cookery. Izaak Walton, “an excellent
angler, and now with God,” dresses chub and trout in his meadow-
sweet pages. Even Thomas Fuller, amid his solacing and delightful
“Worthies,” thinks of the housewife, and gives a receipt for
metheglin.

And a hundred years later Dr. Johnson’s friend, the Rev. Richard
Warner, in his “Personal Recollections,” did not hesitate to expand
upon what he thought the origin of mince pies. Warner’s Johnsonian
weight in telling his fantasy recalls Goldsmith’s quip about the
Doctor’s little fish talking like whales, and also Johnson’s criticism
upon his own “too big words and too many of them.”

Warner wrote, “In the early ages of our country, when its present
widely spread internal trade and retail business were yet in their
infancy, and none of the modern facilities were afforded to the cook
to supply herself ‘on the spur of the moment,’ ... it was the practice
of all prudent housewives, to lay in, at the conclusion of every year
(from some contiguous periodical fair), a stock sufficient for the
ensuing annual consumption, of ... every sweet composition for the
table—such as raisins, currants, citrons, and ‘spices of the best.’

“The ample cupboard ... within the wainscot of the dining parlour
itself ... formed the safe depository of these precious stores.

“‘When merry Christmas-tide came round’ ... the goodly litter of the
cupboard, thus various in kind and aspect, was carefully swept into
one common receptacle; the mingled mass enveloped in pastry and
enclosed within the duly heated oven, from whence ... perfect in
form, colour, odour, flavour and temperament, it smoked, the glory
of the hospitable Christmas board, hailed from every quarter by the
honourable and imperishable denomination of the Mince-Pye.”

In the eighteenth century women themselves, following Hannah

Wolley, began cook-book compiling. So great was their success that
we find Mrs. Elizabeth Moxon’s “English Housewifry” going into its
ninth edition in the London market of 1764. All through history there
have been surprises coming to prejudiced minds out of the despised
and Nazarene. It was so about this matter of cook-books—small in
itself, great in its far-reaching results to the health and development
of the human race.

Women had been taught the alphabet. But the dogmatism of Dr.
Johnson voiced the judgment of many of our forebears: a dominant
power is always hard in its estimate of the capacities it controls.
“Women can spin very well,” said the great Cham, “but they can not
make a good book of cookery.” He was talking to “the swan of
Lichfield,” little Anna Seward, when he said this, and also to a
London publisher. The book they were speaking of had been put
forth by the now famous Mrs. Hannah Glasse, said to be the wife of
a London attorney.

The doctor—possibly with an eye to business, a publisher being

present—was describing a volume he had in mind to make, “a book
upon philosophical principles,” “a better book of cookery than has
ever yet been written.” “Then,” wisely said the dogmatic doctor, “as
you can not make bad meat good, I would tell what is the best
butcher’s meat, the best beef, the best pieces; how to choose young
fowls; the proper seasons of different vegetables; and then how to
roast and boil and compound.” This was the plan of a poet, essayist,
lexicographer, and the leading man of letters of his day. His cook-
book was never written.

But good Mrs. Glasse had also with large spirit aimed at teaching the
ignorant, possibly those of a kind least often thought of by
instructors in her art. She had, forsooth, caught her hare outside her
book, even if she never found him in its page. “If I have not wrote in
the high polite style,” she says, with a heart helpful toward the
misunderstood and oppressed, and possibly with the pages of some
pretentious chef in mind, “I hope I shall be forgiven; for my
intention is to instruct the lower sort, and therefore must treat them
in their own way. For example, when I bid them lard a fowl, if I
should bid them lard with large lardoons, they would not know what
I meant; but when I say they must lard with little pieces of bacon,
they know what I mean. So in many other things in Cookery the
great cooks have such a high way of expressing themselves, that the
poor girls are at a loss to know what they mean.”

Mrs. Glasse’s book was published in 1747—while Dr. Johnson had

still thirty-seven years in which to “boast of the niceness of his
palate,” and spill his food upon his waistcoat. “Whenever,” says
Macaulay, “he was so fortunate as to have near him a hare that had
been kept too long, or a meat pie made with rancid butter, he
gorged himself with such violence that his veins swelled and the
moisture broke out on his forehead.” But within forty-eight years of
the December his poor body was borne from the house behind Fleet
Street to its resting-place in Westminster Abbey, a thin volume, “The
Frugal Housewife,” written by our American Lydia Maria Child, had
passed to its ninth London edition, in that day sales being more
often than in our own a testimony of merit. This prevailing of justice
over prejudice is “too good for any but very honest people,” as Izaak
Walton said of roast pike. Dogmatism is always eating its own
words.

Since the master in literature, Dr. Johnson, planned his cook-book

many cooking men have dipped ink in behalf of instruction in their
art. Such names as Farley, Carême, and Soyer have been written, if
not in marble or bronze, at least in sugar of the last caramel degree
—unappreciated excellencies mainly because of the inattention of
the public to what nourishes it, and lack of the knowledge that the
one who introduces an inexpensive, palatable, and digestible dish
benefits his fellow-men.
The names of these club cooks and royal cooks are not so often
referred to as that of the large and human-hearted Mrs. Glasse. A
key to their impulse toward book-making must, however, have been
that offered by Master Farley, chief cook at the London Tavern, who
wrote in 1791, a hundred and fourteen years ago: “Cookery, like
every other Art, has been moving forward to perfection by slow
Degrees.... And although there are so many Books of this Kind
already published, that one would hardly think there could be
Occasion for another, yet we flatter ourselves, that the Readers of
this Work will find, from a candid Perusal, and an impartial
Comparison, that our Pretensions to the Favour of the Public are not
ill-founded.”

Such considerations as those of Master Farley seem to lead to the

present great output. But nowadays our social conditions and our
intricate and involved household arrangements demand a
specialization of duties. The average old cook-book has become
insufficient. It has evolved into household-directing as well as cook-
directing books, comprehending the whole subject of esoteric
economies. This is a curious enlargement; and one cause, and
result, of it is that the men and women of our domestic corps are
better trained, better equipped with a logical, systematized, scientific
knowledge, that they are in a degree specialists—in a measure as
the engineer of an ocean greyhound is a specialist, or the professor
of mathematics, or the writer of novels is a specialist. And specialists
should have the dignity of special treatment. In this movement, it is
to be hoped, is the wiping out of the social stigma under which
domestic service has so long lain in our country, and a beginning of
the independence of the domestic laborer—that he or she shall
possess himself or herself equally with others—as other free-born
people possess themselves, that is.

And closely allied with this specialization another notable thing has
come about. Science with its microscope has finally taught what
religion with its manifold precepts of humility and humanity has
failed for centuries to accomplish, thus evidencing that true science
and true religion reach one and the same end. There are no menial
duties, science clearly enunciates: the so-called drudgery is often the
most important of work, especially when the worker brings to his
task a large knowledge of its worth in preserving and sweetening
human life, and perfectness as the sole and satisfactory aim. Only
the careless, thriftless workers, the inefficient and possessed with no
zeal for perfection of execution, only these are the menials according
to the genuine teachings of our day—and the ignorant, unlifted
worker’s work is menial (using the word again in its modern English
and not its old Norman-French usage) whatever his employment.

In verse this was said long ago, as the imagination is always

forestalling practical knowledge, and George Herbert, of the
seventeenth century, foreran our science in his “Elixir:”

“All may of thee partake:

Nothing can be so mean,
Which with this tincture for thy sake
Will not grow bright and clean.

“A servant with this clause

Makes drudgery divine;
Who sweeps a room, as for thy laws,
Makes that and th’ action fine.

“This is the famous stone

That turneth all to gold:
For that which God doth touch and own
Cannot for less be told.”

Present-day, up-to-date books on housekeeping stand for the fact

that in our households, whatever the estimates of the past and of
other social conditions, all work is dignified—none is menial. For
besides intelligent knowledge and execution, what in reality, they
ask, gives dignity to labor? Weight and importance of that particular
task to our fellow-beings? What then shall we say of the duties of
cook? of housemaid? of chambermaid? of the handy man, or of the
modest maid of all work? For upon the efficient performance of the
supposedly humblest domestic servitor depends each life of the
family. Such interdependence brings the employed very close to the
employer, and no bond could knit the varied elements of a
household more closely, none should knit it more humanly.

The human, then, are the first of the relations that exist between
employer and employee, that “God hath made of one blood all
nations of the earth.” It is a truth not often enough in the minds of
the parties to a domestic-service compact. And besides this gospel
of Paul are two catch-phrases, not so illuminated but equally
humane, which sprang from the ameliorating spirit of the last
century—“Put yourself in his place,” and “Everybody is as good as I.”
These form the best bed-rock for all relations between master and
servant. There is need of emphasizing this point in our books on
affairs of the house, for a majority of our notably rich are new to
riches and new to knowledge, and as employers have not learned
the limitation of every child of indulgence and also polite manners in
early life.

It is after all a difference of environment that makes the difference

between mistress and maid, between master and man. The human
being is as plastic as clay—is clay in the hands of circumstance. If
his support of wife and children depended upon obsequiousness of
bearing, the master might, like the butler, approximate Uriah Heep.
If the mistress’s love of delicacy and color had not been cultivated by
association with taste from childhood, her finery might be as vulgar
as the maid’s which provokes her satire. It is after all a question of
surroundings and education. And in this country, where Aladdin-
fortunes spring into being by the rubbing of a lamp—where families
of, for example, many centuries of the downtrodden life of European
peasant jump from direst poverty to untold wealth—environment has
often no opportunity to form the folk of gentle breeding. Many
instances are not lacking where those who wait are more gently
bred than those who are waited upon.

In their larger discourse, then, up-to-date household books stand for

the very essence of democracy and human-heartedness—which is
also the very essence of aristocracy. After the old manner which
Master Farley described, our women seem to have given their books
to the public with the faith that they contain much other books have
not touched—to stand for an absolutely equable humanity, for
kindness and enduring courtesy between those who employ and
those who are employed, the poor rich and the rich poor, the
householders and the houseworkers—to state the relations between
master and man and mistress and maid more explicitly than they
have before been stated, and thus to help toward a more perfect
organization of the forces that carry on our households—to direct
with scientific and economic prevision the food of the house
members; to emphasize in all departments of the house
thoroughgoing sanitation and scientific cleanliness.

Of questions of the household—of housekeeping and home-making

—our American women have been supposed somewhat careless.
Possibly this judgment over the sea has been builded upon our
women’s vivacity, and a subtle intellectual force they possess, and
also from their interest in affairs at large, and again from their
careful and cleanly attention to their person—“they keep their teeth
too clean,” says a much-read French author. Noting such
characteristics, foreigners have jumped to the conclusion that
American women are not skilled in works within doors. In almost
every European country this is common report. “We German women
are such devoted housekeepers,” said the wife of an eminent
Deutscher, “and you American women know so little about such
things!” “Bless your heart!” I exclaimed—or if not just that then its
German equivalent—thinking of the perfectly kept homes from the
rocks and pines of Maine to the California surf; “you German women
with your little haushaltungen, heating your rooms with porcelain
stoves, and your frequent reversion in meals to the simplicity of
wurst and beer, have no conception of the size and complexity of
American households and the executive capabilities necessary to
keep them in orderly work. Yours is mere doll’s housekeeping—no
furnaces, no hot water, no electricity, no elevators, no telephone,
and no elaborate menus.”

Our American women are model housekeepers and home-makers, as

thousands of homes testify, but the interests of the mistresses of
these houses are broader, their lives are commonly more projected
into the outer world of organized philanthropy and art than women’s
lives abroad, and the apparent non-intrusion of domestic affairs
leads foreigners to misinterpret their interest and their zeal. It is the
consummate executive who can set aside most personal cares and
take on others efficiently. Moreover, it is not here as where a learned
professor declared: “Die erste Tugend eines Weibes ist die
Sparsamkeit.”

To have a home in which daily duties move without noise and as like
a clock as its human machinery will permit, and to have a table of
simplicity and excellence, is worth a pleasure-giving ambition and a
womanly ambition. It is to bring, in current critical phrase, three-
fourths of the comfort of life to those whose lives are joined to the
mistress of such a household—the loaf-giver who spends her brains
for each ordered day and meal. Moreover, and greatest of all, to plan
and carry on so excellent an establishment is far-reaching upon all
men. It is the very essence of morality—is duty—i.e., service—and
law.

The French aver that men of the larger capacity have for food a
particularly keen enjoyment. Possibly this holds good for Frenchmen
—for the author of Monte Cristo, or for a Brillat-Savarin, of whose
taste the following story is told: “Halting one day at Sens, when on
his way to Lyons, Savarin sent, according to his invariable custom,
for the cook, and asked what he could have for dinner. ‘Little
enough,’ was the reply. ‘But let us see,’ retorted Savarin; ‘let us go
into the kitchen and talk the matter over.’ There he found four
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

textbookfull.com